NAME

mpxstut - Building Shared Libraries for MacPerl


SYNOPSIS

[ MPW (Codewarrior or Apple compilers/linkers) ]
   h2xs -A -n Example1

   Directory Example1

   ...write XS and Perl code...

   perl Makefile.PL

   BuildProgram all|dynamic

   ...test...

   BuildProgram install

[ Codewarrior IDE ]
    As described below


DESCRIPTION

This tutorial discusses the process of building extension modules, which incorporate shared libraries, for MacPerl. We will examine the use of the Codewarrior compilers and linkers, either in the Codewarrior IDE or in Codewarrior MPW, and also how to use the compilers and linkers available from Apple - MrC and SC, PPCLink and ILink.

Extensions for Perl mean writing C code, plain and simple. It may be that the functionality you want is already available in C, or you want to use C for speed. Ultimately the C source is exposed as a Perl API (application programming interface) consisting quite often of a one-to-one translation of C functions to Perl builtins.

There are four phases to building an extension for Perl:

    (1) Converting so-called XS code to C;

    (2) Compiling the C source;

    (3) Linking the object code into a shared library;

    (4) Editing your Perl interface.

As a MacPerl extension developer you can avail yourself of the basic utilities for producing extension file templates, and for installing your finished libraries.

It almost goes without saying, but if you are going to seriously build (and create) MacPerl extensions, you should be comfortable with C, as well as with all of the development tools for the environment you choose to use.


A SHORT HISTORY

The original motivation for this tutorial is, in a sense, over a year old. At the time I was working with the Mac Toolbox extensions, and the distribution version of Mac::Controls was flawed in several respects. To make a long story short, I needed a new Controls library, I wasn't even aware of the MPW build procedures described in MPPE (MacPerl: Power and Ease, by Chris Nandor and Vicki Brown), and so I reverse engineered the build rules files.

One thing that immediately came up was the library issue. The existing procedure required libraries that I simply did not have as the proud possessor of (at the time) Codewarrior Pro 2. So part of my tweaking involved figuring out how to build extensions for MacPerl with libraries at hand.

Things have evolved from there...

As of this writing (May 1999), the use of CW Pro 2 or 3 (and presumably earlier and later versions of same) to build basically any CPAN XS-based module, XS-based modules of your own, or Toolbox modules, is a well-established procedure. There is more baseline experience with Codewarrior MPW, but using only the Codewarrior IDE also appears to be reliable. It also appears, although there is less of a track record, that building MacPerl extensions for PPC using Apple MPW is also robust. The procedures for CFM68K with Apple MPW are at the developmental stage.


TERMINOLOGY

Shared Libraries
For shorthand, this tutorial will occasionally refer to shared libraries as DLL's (dynamically-loaded libraries).

module
A module will normally mean the entire set of files that makes up a proper CPAN module distribution. From time to time the word module may also just refer to the .pm file with the same basename.

extension
We're interested in extensions, and for our purposes here an extension module contains at least one source file with C code. Use of the terms extension and extension module is interchangeable.

XS files
Writing the C source is simplified if you prepare a file using XS macros, and run xsubpp on it to write a C source file. A reference to an .xs file means this initial file that must have xsubpp run on it. If you feel bold, you may always write the C file directly.

Source Tree
When you download the MacPerl source distribution, it unpacks as a folder MacPerl_Src. You may elect to put it in your MacPerl application folder, or keep it separate - in any case the MacPerl_Src folder will be referred to as containing the source tree for MacPerl. In other words, the hierarchy of source and header files pertaining to MacPerl. Note that the folder MacPerl_Src:perl contains the files that you expect to find in a standard Perl distribution, although there are extras, and some standard files have been modified.


DOCUMENTATION AND REFERENCES


MPW

1. Building and Managing Projects with MPW in PDF format. Helpful for Codewarrior MPW; indispensable for using Apple MPW. Available at ftp://dev.apple.com/developer/Tool_Chest/Core_Mac_OS_Tools/MPW_etc./.

2. The MPW online help. For example, any of the following commands:

    Help MrC
    Help MrCpp
    Help PPCLink
    Help SC
    Help SCpp
    Help ILink
    Help MakeFlat

    Help MWCPPC -f "{MPW}"Metrowerks.Help
    Help MWLinkPPC -f "{MPW}"Metrowerks.Help
    Help MWC68K -f "{MPW}"Metrowerks.Help
    Help MWLink68K -f "{MPW}"Metrowerks.Help

    Help DumpPEF
    Help DumpObj

3. The dmake.nc manual page in the man folder in your dmake distribution.


Codewarrior IDE

Target MacOS, C Compiler Ref, MSL C Ref in PDF or Quickview format. Available on the CW Reference CD.


General

1. All the Perl manpages pertaining to Perl internals and XS, namely perlxstut, perlxs, perlguts. PerlGuts Illustrated by Gisle Aas is helpful. So is perlcall and perlio.

2. Dean Roehrich's XS Cookbooks on CPAN are very useful if you're looking to improve your knowledge of XS. These may be found at http://www.perl.com/CPAN//authors/Dean_Roehrich/.

3. For module building, the POD for h2xs and xsubpp. Also, the Perl Cookbook by Tom Christiansen and Nathan Torkington (highly recommended).

4. MacPerl:Power and Ease, by Chris Nandor and Vicki Brown. In particular, the chapter describing the standard procedure for building extensions with Codewarrior MPW. To order, or to download HTML of parts of the book, visit the Prime Time Freeware site at http://www.ptf.com/.


REQUIREMENTS

In order to make use of material in this tutorial, you must have a C/C++ compiler for your platform. This means either MrC/MrCpp and SC/SCpp with Apple MPW, or Codewarrior (with or without Codewarrior MPW). The Codewarrior referred to here is the Professional version, not the ``Discover Programming'' version.

The MPW perl tool must be properly installed if you choose to use MPW. See the file Install.MPW_Perl in the basic MacPerl distribution.

You must have the MacPerl source distribution.

You must have dmake, by Dennis Vadura. Note: you want the Matthias Neeracher port (available from his site), not any of the distributions from the dmake website. One source is ftp://sunsite.cnlab-switch.ch/software/platform/macos/src/mpw_c/dmake_40m2.sit.bin.


PREPARING THE SOURCES


Multi-Track Tutorial

If you're using MPW (Apple or Codewarrior), the best place to build is in the :MacPerl_Src:perl:ext folder, or a subfolder thereof. Use 'Set  Directory' to make this the working directory. xsubpp looks for the typemap in :MacPerl_Src:perl:lib:ExtUtils, and if you're doing Toolbox stuff, also may search for the typemap in :MacPerl_Src:perl:ext:Mac.

If you're using Codewarrior (the IDE), it doesn't really matter where you build your project. However, if you choose to avail yourself of the modified h2xs and xsubpp utilities supplied in the Mac_XS distribution, you must build in the MacPerl source tree, as above, or explicitly specify the path to the :MacPerl_Src:perl:lib:ExtUtils typemap when running xsubpp.

Special Note: where a ¶ appears in scripts, that is meant to be an Option-D.


Where To Start

A shared library for MacPerl is built around an XS file, and perhaps some C source files and header files. xsubpp is used to compile the XS into a valid C file.

If you are writing your own extension from scratch, read about h2xs. If not, we're likely talking about a CPAN distribution, and you may skip the discussion of h2xs in the tutorial track appropriate for your development environment.


h2xs

The primary purpose of h2xs, as the name suggests, is the production of XS files based on C header files. These header files describe the API to some external C library which is the core of the intended DLL.

h2xs constructs a complete set of files for a proper CPAN-conforming Perl (extension) module. If no C header files are supplied, the skeleton is there nonetheless. It is strongly recommended that you use h2xs when writing new modules, even if they contain no XS. You will be less likely to make mistakes.

Normally we run

    perl h2xs.PL

to extract h2xs, and put h2xs into an MPW command path, e.g. :MPW:Scripts:. However, we've modified h2xs to better handle nested packages; the modified script is available in the Mac_XS distribution on The MacPerl Pages. Place this into an MPW command path.

Afterwards, if creating a distribution from scratch, try something like

    h2xs -A -n Examples::ExampleA

Note: For parsing C headers and automatically generating XS declarations, h2xs needs to have a functional C::Scan module, and right now it's not. So specifying a header file is at your own risk. If you want to interface a C library as fast as possible, you may wish to consider SWIG. A short tutorial on building MacPerl extensions with SWIG instead of XS will be available shortly as the companion to this work.


BUILDING WITH APPLE MPW

When we refer to using Apple MPW, we mean not only Apple MPW but also only the free compilers - MrC/MrCpp for PPC, and SC/SCpp for 68K, along with the associated linkers and other tools. In other words, this is the free alternative to using Codewarrior.


MacPPC

The following MPW script, courtesy of Alan Fry, and somewhat modified by myself, is a stand-alone, which may be used as is. It serves to show you the barebones of what is really going on:

    #   Write module and XS files e.g. myModule.pm, myModule.xs 
    #   both in a folder with the same name i.e. myModule.
    #   Change into the folder myModule and run the following
    #   commands.
    
    Set name "myModule"
    Set MPSrc "Dynatek1:MacPerl_Src"
    
    # Add extra typemaps here using the -typemap option to xsubpp.
    # Look at 'xsubpp' to see which ones
    # are picked up as a matter of course

    perl  {MPSrc}:perl:lib:ExtUtils:xsubpp -prototypes ¶
           "{name}".xs > xstmp.c
           
    Rename -y xstmp.c "{name}".c

    # To be honest, I can't remember what build needed the
    # __STD_C #define. You can likely leave this out (?)

    MrC -inclpath ignoresys -sym on -d __STD_C -d MULTIPLICITY ¶
         -w off -opt size -t -fatext ¶
        -i {MPSrc}:sfio:include: ¶
        -i {MPSrc}:GUSI:include: ¶
        -i {MPSrc}:perl: ¶
        -i : ¶
        -shared_lib_export on -export_list "{name}".exp ¶
        "{name}".c
    
    # edit the '{name}.exp' file to retain only the
    # 'boot_XXX' symbol

    PPCLink -w -xm sharedlibrary -sym on ¶
        -@export "{name}".exp ¶
        "{name}".c.PPC.o ¶
        {SharedLibraries}InterfaceLib ¶
        {SharedLibraries}StdCLib ¶
        {SharedLibraries}MathLib ¶
        {MPSrc}:perl:PerlStub ¶
        -c cfmg ¶
        -fragname "{name}" ¶
        -o "{name}".shlb.PPC

You can cut and paste the commands in one-by-one; execute them by selecting them and pressing the Enter key.


MacCFM68K

The build procedure for CFM68K is quite similar; however, there is some extra preparation required of the source file(s).

You must bracket your declarations in the XS file with appropriate #import pragmas. Here's an example:

    #include <ConditionalMacros.h>
    #if PRAGMA_IMPORT
    #pragma import on
    #endif
    
    #include "EXTERN.h"
    #include "perl.h"
    #include "XSUB.h"
    
    /* these will vary - see the discussion */
    OSErr Path2FSSpec(const char * path, FSSpec * desc);
    void CopyC2PStr(char * cstr, StringPtr pstr);

    #if PRAGMA_IMPORT
    #pragma import off
    #endif

Some (not much) explanation is in order. By declaring symbols (routines and variables) as imports, the compiler treats them quite differently, and so at link time the linker doesn't try to find various Perl routines in the object file you just built.

The two explicit declarations above (for Path2FSSpec and CopyC2PStr) won't be needed at all unless you build a Toolbox extension. And then, depending on which Toolbox extension you build, they will differ. For example, the two functions declared above pertain specifically to building Mac::Resources. What's going on is that the Toolbox code is using a variety of Mac types, and the typemap HD:MacPerl_Src:perl:ext:Mac:typemap (which is worth taking a look at) calls a variety of Toolbox routines to convert these types in the XS code (strictly speaking, xsubpp uses the typemap). So these routines need to be declared as imports also.

The easiest way to find out what to explicitly declare in this manner is to declare nothing, let the linker tell you what it can't find, and then use a batch search in Alpha or BBEdit or MPW to locate the appropriate declaration (which will likely be in a file in the perl folder).

If you have a distribution that also contains a number of C header files and C source files, there may be declarations in a variety of places. These probably require the same treatment. It's more convenient to wrap the declarations in the header file than it is to do so in each source file where the header is #included. It's easier to do than it is to explain, believe me. Recommendation: start off with the simplest cases, and develop a feel for basic CFM68K build procedures with SC and ILink, before tackling something like SQL::Statement. The Mac_XS distribution does contain an MPW worksheet for XML::Parser, though.

The actual script for CFM68K, once you've prepped your XS, is as follows: # Write module and XS files e.g. myModule.pm, myModule.xs # both in a folder with the same name i.e. myModule. # Change into the folder myModule and run the following # commands. # I hope you get the drift of the 'firstPart' and 'secondPart' # For example, for the module SQL::Statement, 'name' becomes # ``Statement'' and 'boot_sym' is ``boot_SQL__Statement''

    Set name "secondPart"
    Set boot_sym "boot_firstPart__secondPart"
    Set MPSrc "Dynatek1:MacPerl_Src"
    
    # Add extra typemaps here using the -typemap option to xsubpp.
    # Look at 'xsubpp' to see which ones
    # are picked up as a matter of course

    perl   {MPSrc}:perl:lib:ExtUtils:xsubpp -noprototypes ¶
           {name}.xs > xstmp.c
    Rename -y xstmp.c {name}.c
    
    # use '-opt parameter' to control global optimizations
    # '-model cfmFlat' also sets '-mc68020' and '-bigseg'

    SC -w off -d __STD_C -d MULTIPLICITY ¶
        -model cfmFlat ¶
        -i {MPSrc}:sfio:include: ¶
        -i {MPSrc}:GUSI:include: ¶
        -i {MPSrc}:perl: ¶
        -i : ¶
    -o {name}.c.o {name}.c
    
    ILink ¶
        {name}.c.o ¶
        -w ¶
        -xm sharedlibrary ¶
        -model cfmflat ¶
        {SharedLibraries}InterfaceLib ¶
        {SharedLibraries}StdCLib ¶
        {SharedLibraries}MathLib ¶
        {MPSrc}:perl:PerlStub ¶
        -c cfmg -t shlb -fragname {name} ¶
        -export {boot_sym} ¶
        -o "{name}".shlb.CFM68K
    
    SetFile -t 'shlb' "{name}".shlb.CFM68K

Look at your XS file to see what the identifier of the boot routine is. Again, just use this as a template (copying and pasting into your MPW worksheet), or save it as a file, and execute it by typing its name and pressing Enter.


Testing and Installation

With reference to the INSTALLATION ISSUES section below, either set up a local installation, say in a folder called lib, or manually install into an existing library search path. Please refer to the TESTING AND VERIFICATION section below for more details.


BUILDING WITH CODEWARRIOR MPW


An Overview

The build process in MPW is designed to follow, as closely as possible, the standard UNIX procedures for building extensions for Perl. So this discussion, by definition, resorts in large measure to telling you to invoke a sequence of coomands. These may appear like black boxes, and if an error occurs, or something is not configured quite right, there is potential for confusion.

In the interests of demystifying what the overall procedures do, here is an edited template, similar to that for Apple MPW, which shows you what happens for a typical module, right up to the point before things get installed.

    # this is where the object files go
    NewFolder Obj
    
    Set MPSrc "Tallinn:MacPerl_Src"
    Set Name "name"
    
    # This full set of typemaps is applicable to a Toolbox build.
    # Edit accordingly
    
    perl -I {MPSrc}:perl:lib: {MPSrc}:perl:lib:ExtUtils:xsubpp ¶
    -typemap {MPSrc}:perl:ext:Mac:QuickDraw:typemap ¶
    -typemap {MPSrc}:perl:ext:Mac:Controls:typemap ¶
    -typemap {MPSrc}:perl:ext:Mac:Events:typemap ¶
    -noprototypes {Name}.xs > xstmp.c
    
    Rename -y xstmp.c :{Name}.c
    
    # Put in a list of your source files here
    Set SrcPPC ":{Name}.c"
    
    MWCPPC -nosyspath -sym on -d MULTIPLICITY -w nounusedarg ¶
    -i- -i : -i "{MPSrc}:sfio:"include: ¶
    -i "{MPSrc}:GUSI:"include: ¶
    -i "{MWCIncludes}" -i {MPSrc}:perl: ¶
    -w nopossible -opt all -t -ext .PPC.o {SrcPPC} -o :Obj:
    
    # You don't have to use this, but if you don't, write a
    # file called "{Name}.exp" which has a single line, of
    # form 'boot_MyModule'. Take a look at the XS file to see
    # what the name of the boot function is.
    
    Set Module "myModule"
    # Example: "Mac::Dialogs"
    
    perl "-I {MPSrc}:perl:lib:" -e 'use ExtUtils::Mksymlists; ¶
        Mksymlists("NAME" => shift, ¶
        "DL_FUNCS" => {  }, "DL_VARS" => []);' 
{Module}
    
    # Linking. Modify as required for extra object files or
    # user libraries
    
    MWLinkPPC -xm sharedlibrary -sym on -msg nodup ¶
    -@export {Name}.exp -name {Name} ¶
    -o {Name}.shlb.PPC :Obj:{Name}.c.PPC.o ¶
    "{MWPPCLibraries}MathLib" ¶
    "{MWPPCLibraries}InterfaceLib" ¶
    "{MWPPCLibraries}MSL RuntimePPC.Lib" ¶
    "{MWPPCLibraries}MSL C.PPC.Lib" ¶
     {MPSrc}:perl:PerlStub
    
    # The same procedures for CFM68K
    
    Set SrcCFM68K ":{Name}.c"
    
    MWC68K -nosyspath -sym on -d MULTIPLICITY -w nounusedarg ¶
    -i- -i : -i "{MPSrc}:sfio:"include: ¶
    -i "{MPSrc}:GUSI:"include: ¶
    -i "{MWCIncludes}"  -i {MPSrc}:perl: ¶
    -w nopossible -indirect -mc68020 -model CFMflatdf ¶
    -pragma "export on" -model far -opt off -t ¶
    -ext .CFM68K.o {SrcCFM68K} -o :Obj:
    
    MWLink68K -xm sharedlibrary -sym on -msg nodup ¶
    -@export {Name}.exp -name {Name} ¶
    -o {Name}.shlb.CFM68K :Obj:{Name}.c.CFM68K.o ¶
    "{MW68KLibraries}MathLibCFM68K Fa(4i_8d).Lib" ¶
    "{MW68KLibraries}InterfaceLib" ¶
    "{MW68KLibraries}MSL C.CFM68K Fa(4i_8d).Lib" ¶
    "{MW68KLibraries}MSL MWRuntimeLibCFM68K" ¶
    "{MW68KLibraries}MSL ShLibRuntimeCFM68K.Lib" ¶
     {MPSrc}:perl:PerlStub

If you have problems editing your Build Rules files, or installing dmake, or just want simplicity, use this script as a template - copy and paste it into an MPW worksheet, Set Directory to your module folder, and off you go!

Note: you should still read the section on EDITING THE BUILD RULES, particularly the part on libraries, as we cannot specify a universal set of libraries for all possible environments. If there are modifications to be made to the above, this is likely where you will make them.


Creating the Makefile

Start here if you're going to follow the MPPE process.

The XS build process in MPW is driven by a makefile, just as it is on Unix. However, the make we use here is not MPW Make, but rather dmake, which is a much more conventional make.

We start by running the command

    perl Makefile.PL

This creates a dmake makefile called Makefile.mk. It also generates an Obj folder.

Note: once you've got Makefile.mk, you can generate the MPW build commands, as in the script above, for the specific extension, by using the procedure described under DIAGNOSTICS:MPW Build Hacks below. Use this method if you anticipate problems, or just feel more comfortable running commands one by one.

You should get into the habit of reviewing this file (Makefile.mk) as a matter of course. For example, if you haven't built MacPerl itself, you won't have a miniperl. What I routinely do is edit both the PERL and FULLPERL variables to

    PERL = perl
    FULLPERL = perl

There are a number of other ``gotchas'', and some are covered in the Diagnostics section. For distributions with C source and header files, possibly in sub-folders, you should review all the path information, and edit as required.

If you're running these procedures for the first time, jump to the section Editing the Build Rules. If not, proceed to Compiling and Linking.


Compiling and Linking

We're not interested in static libraries, so we can start the build at a more specific target:

    BuildProgram dynamic

This will chug away, and you should be able to track the compile and link cycles for both PPC and CFM68K. The shared libraries are placed in the Obj folder.


Local Installation

Issue the MPW command

    BuildProgram install_dynamic

This does 2 things. One, all .pm files and shared libraries, and any .al files created by Autosplit, are installed in a local folder named :blib:lib. This is handy for testing.

Two, the same files are duplicated in the :perl:lib folder in the MacPerl source tree. This latter behaviour is Unixish, and not so helpful to us if our MacPerlPerl application library search paths are to :lib and :site_perl in the MacPerl application folder.

The main point here, if you are not familiar with Unix, is that after the above command the files are still not in the final desired location.

Note: BuildProgram has a primary function of choosing and running a make for us. We're using dmake. Since dmake is a make like any other, you could, if you so wished start with the install_dynamic target as opposed to dynamic; dmake would detect that objects haven't yet been built and would proceed to do so.


Testing

See the section on Testing and Verification, below.


Final Installation

Manually install (refer to the notes in the INSTALLATION ISSUES section above), or even better, use Chris Nandor's excellent little installme droplet. (There are some gotchas to be aware of when using the latter - see Diagnostics).


BUILDING WITH THE CODEWARRIOR IDE


h2xs

The Mac_XS distribution, available from The MacPerl Pages, contains a modified h2xs script called mac_h2xs, and a wrapper GUI front-end called h2xs_app.plx, which allow for generation of the more common types of extensions.

Run h2xs_app.plx just like any other script. The dialog allows you to set the -A, and -h switches to h2xs. You also must specify a module name, and it is recommended that you specify the build folder. Under normal circumstances you should build your extensions in the MacPerl source tree.


xsubpp

This we need to run separately to turn XS code into C, unless you're into writing the C yourself. This is doable, but why re-invent the wheel? The Mac_XS distribution contains a droplet called xsubpp_app; when you're happy with your XS file, drag it onto the droplet. It will generate a C source file in the same directory.


Setting up your Project

As an example, let's try XML-Parser (I'm doing version 2.23). Put the distribution into Your HD:MacPerl_Src:perl:ext:, and unpack with Chris Nandor's untarzipme (if you don't have his CPAN utilities, use Stuffit Expander, but bear in mind that each of the files will have to have line-ends converted from Unix to Mac).

Enter the XML-Parser folder. There is one .xs file, Expat.xs in folder Expat. Edit this to change line 30 to

#define BUFSIZE 26624

This is so we don't exceed a 32K static data limit.

Drag Expat.xs onto the xsubpp_app droplet, which produces the C source file Expat.c. Start off by choosing New Project, and select MacOS, C_C++, Standard Console, CFM68K or PPC as appropriate. We're going to make enough changes that this initial choice is non-critical. Add Expat.c to your project.

You will also want to #define the macro MULTIPLICITY. This is used by embed.h; the MPW build process adds it by default. You can edit the main XS file to put this #define in before any of the #include's. A better approach is to write a little header file that contains this #define, which you will #include as a prefix file under C/C++ Language in the Project Settings.

Here's where looking at Makefile.PL still comes in handy even with Codewarrior. This file, the one in folder Expat, not the top one, will let you know to also add the files in the following statement:

    @expat_files = qw(expat/xmltok/xmltok.c expat/xmltok/xmlrole.c
        expat/xmlparse/xmlparse.c expat/xmlparse/hashtable.c);

Following that, create a new group, call it PerlLib or something similar, and add PerlStub.

Open the Project Settings, and start with Access Paths. Note that by adding PerlStub we already have one required path. Now we need to add :sfio:include and :GUSI:include to the System paths. Make sure they precede the system headers.

Just like previously, looking at :Expat:Makefile.PL lets us know to add two other access paths as User Paths:

    INC       => "-Iexpat/xmltok -Iexpat/xmlparse"

Under Target select Shared Library. Give it a name like XML_Parser_PPC or XML_Parser_CFM68K.

Under C/C++ Language, uncheck ``Require Function Prototypes''. Also, if your C code is studded with \n and \r escapes, you likely want to check the ``Map newlines to CR'' checkbox. Note that this option is the default when using MPW. For more on this see the section on Newlines. We will do this for this library - often it doesn't matter.


PPC Processor and Linker Options

Code Generation > PPC Processor
No changes for the purpose of this tutorial. In practise I do some optimizations.

Linker > PPC Linker
Normal Linking, generate a Link Map (as desired), __initialize for the Initialization entry point, nothing for Main, __terminate for the Termination entry point.

Linker > PPC PEF
``Use .exp file'' for Export Symbols.


CFM68K Processor and Linker Options

Processor > 68K Processor
Large code model, 68K alignment, SANE. 68020 code generation. No global optimization, all ``far'' options checked, MPW C calling conventions checked.

Linker > 68K Linker
Check everything under Linker Info except Fast Linking. I also ask to generate a Link Map, at least at the ``What?'' stage of things.

Linker > CFM68K
``Use .exp file'' for Export Symbols. Share Data, and Force Indirect Access. Entry points __initialize and __terminate (no entry point for Main).


Getting the Right Libraries

When you select a C/C++ MacOS Std Console Project for PPC you actually get all the right libraries, with the exception of PerlStub.

With CFM68K you don't. What you want to have in place for the most general case is a CFM C library, a CFM Math library, a CFM Runtime library, and I've found that even for the monolithic shared libraries it's nice to have the ShLibRuntime library for CFM also. Furthermore you want the fat InterfaceLib. These are all easy enough to find in the MacOS Support and Standard Libraries folders in the Codewarrior folder.

Take a look at the MPW discussion in the Editing the Build Rules section (in this tutorial) for tips. Also, read the Target MacOS.pdf manual in the CW docs carefully. Understanding what the libraries are doing is important to becoming comfortable with these builds.


Compiling

Select the source file, and run Compile. The first time you do this, Codewarrior will complain that errno is being redefined from long to int. So edit perl.h and make this declaration a long. You will also have to do this in sys/errno.h. This is a hack - I'm unapologetic.

There will be a complaint about line 68 in icemalloc.h:

#endif DEVELOPMENT

so edit this to remove the ``DEVELOPMENT''. Subsequently both files will compile with some warnings.

Now ``Make'', and the library will be built. You'll see a large number of warnings - this has to do with the fact that umpteen symbols are defined in PerlStub that are defined in the other libraries.

Open the .exp file, and edit to retain only the line

    boot_XML__Parser__Expat

Rerun the ``Make''. You'll note that the shared library shrinks in size considerably.


Testing and Installation

That's it! Manually set up a local folder structure for testing, and run your tests using the MacPerl application. See the section on Testing and Verification. Once you're content, install manually or use Chris Nandor's installme script. Note: if you're using the latter, manually set up a proper local :blib:lib folder structure.


INSTALLATION ISSUES

This applies to building in all three development environments.

Whether you start with a CPAN distribution, or with a folder created by h2xs, you'll end up with one or more shared libraries (two if you're creating for both PPC and CFM68K), the associated .pm files, and if Autosplit was required, one or more .al files. All of which will be in this original folder, which we'll call the distribution folder.

A local installation will be a folder containing all of the above files, organized in such a fashion that you can add this folder to your MacPerl (app or tool) library paths, and execute scripts which use the new module. This folder will be in the distribution folder. This is useful for testing and development. The standard Codewarrior MPW build procedure constructs such a local installation in a folder called blib.

For general use, however, we want to take the important files, those mentioned above, and put them in a standard location, that is, a location specified by a regular MacPerl library search path.

These locations are specified as follows:

MacPerl app
The Libraries field in the Edit -> Preferences menu. Typically something like

    MyHD:MacPerl Ä:lib:
    MyHD:MacPerl Ä:site_perl:

MPW perl tool
The PERL5LIB environment variable in the Perl startup file in :MPW:Startup Items:. Usually the same as for the application.

Whatever the search path, local or general, for an arbitrary module, say ExampleA::Utils, this is the full structure that starts at the folder pointed to by that path:

    ExampleA:
        Utils.pm
        Utils.xs (include this if the XS file contains POD)

    MacPPC:
        auto:
            ExampleA:
                Utils:
                    Utils (the PPC shared library)

    MacCFM68K:
        auto:
            ExampleA:
                Utils:
                    Utils (the CFM68K shared library)
    auto:
        ExampleA:
            Utils:
                *.al (files produced by Autosplit - OPTIONAL)

This is the structure you will see when you look at the :blib:lib: folder in your distribution folder (which is produced during the Codewarrior MPW build process), this is the structure yu will see if you look at :lib: or :site_perl: in your MacPerl application folder, and so on and so forth. It is good to be familiar with it.


NOTES ON TESTING AND VERIFICATION


Testing

Well-behaved CPAN distributions contain either a test.pl file or a bunch of *.t test scripts in a t directory. h2xs generates a skeleton test.pl file when you run it.

The general UNIX procedure, once a blib folder is in place (previous section), is to run

    make test

This makes use of the Test::Harness module, which out-of-the-box doesn't work on Macs. The upshot is that we can't use the standard procedure.

For testing with MPW, use

    perl -I ':blib:lib:' test.pl

or

    For tfile In `Files t`
        Echo {tfile}
        perl -I ':blib:lib' :t:{tfile}
    End

For the MacPerl application, we'll be in the t folder. Usually the .t files will assume that the working directory is the build folder (the parent of the t folder); therefore path references will be off by one, and anticipate doing some edits for this. Under Edit > Preferences, put in the path to the folder containing your library, ensuring that it's ahead of :site_perl: if you have an older version of the same library already on your system. Then just run the scripts one by one.

Note: It's not terribly difficult to hack Test::Harness so as to retain some of the functionality - it's a matter of using require or do on the .t files rather than a piped open. The main problem is that .t files not infrequently terminate with an exit statement, which of course shuts down the whole procedure. Hence the above.


Verification

A good habit to acquire is checking your libraries to see what symbols they export, and where they import symbols from.

When building libraries with the Codewarrior IDE, check Use Link Map under Linker options. The result is a text file. The Import Containers sections near the end tell you where symbols are being referenced from. If you suspect that a symbol is being imported from the wrong library, you can find out here.

In MPW use DumpPEF. It's as simple as

    DumpPEF :path_to_lib:libName

The section entitled Loader Import Symbol Table tells you where symbols are being imported from (what other library), and the section near the end entitled Loader Export Symbol Table tells you what's being exported.

Near the beginning of the DumpPEF listing you'll also notice a line similar to

    Container [1] (Dialogs) ...

which tells you the name of your fragment (library).


EDITING THE BUILD RULES

Read this section if you want to be able to use the procedures described in MacPerl:Power and Ease. If you're happy with using the worksheet templates, you should still read the parts concerning libraries and linking.

This applies mostly to Codewarrior MPW development. However, you can certainly use the standard build procedure with the MrC compiler and the PPCLink linker available in Apple MPW - it's a matter of editing the build rules files just as described here, making appropriate changes not only to the libraries but also to variables such as CPPC and LinkPPC.

When a Makefile.mk file is generated, 3 other files are included using dmake .INCLUDE directives. These files contain all of the common build rules, and as such will not have to be edited from one XS build to another.

The first time around, you will need to edit two of these files.


BuildRules.mk

Comment out lines

    OldMW68KLibraries   = $(MPW)Libraries:OldMW68KLibraries:
    OldMWPPCLibraries   = $(MPW)Libraries:OldMWPPCLibraries:

because we don't have them. Put in

    OldMW68KLibraries   = 
    OldMWPPCLibraries   = 

instead. Comment out ANSI, and put in something like

    CWMPW   = Tallinn:Metrowerks:CodeWarrior¶ MPW
    ANSI    = $(CWMPW):Interfaces&Libraries:Interfaces:MWCIncludes:

Note that there is a defined dmake $(MPW) variable, but it points to the folder called MPW enclosed in folder CodeWarrior MPW.


ExtBuildRules.mk

    PERL     = $(PERL_SRC)miniperl

to

    PERL    = perl

and replace the library specs to something like

    DYNAMIC_STDLIBS_PPC *= \
        "{{MWPPCLibraries}}MathLib" \
        $(PERL_SRC)PerlStub \
        "{{MWPPCLibraries}}InterfaceLib" \
        "{{MWPPCLibraries}}MSL RuntimePPC.Lib" \
        "{{MWPPCLibraries}}MSL C.PPC.Lib"
    
    DYNAMIC_STDLIBS_CFM68K *= \
        "{{MW68KLibraries}}MathLibCFM68K Fa(4i_8d).Lib" \
        $(PERL_SRC)PerlStub \
        "{{MW68KLibraries}}InterfaceLib" \
        "{{MW68KLibraries}}MSL C.CFM68K Fa(4i_8d).Lib" \
        "{{MW68KLibraries}}MSL MWRuntimeLibCFM68K" \
        "{{MW68KLibraries}}MSL ShLibRuntimeCFM68K.Lib"

Getting the right libraries is the key to making this whole process work. I am no MacOS expert, and quite honestly these libraries were arrived at with lots of reading and even more BASWBO (Build And See What Blows Up).

So this list is provisional. It's for Codewarrior; for Apple MPW see the script in the Building with Apple MPW section above. The main point to be made is that the various tools in the development cycle - the compiler, the linker, various diagnostic utilities - will steer you to the correct header files, options and libraries. That's why it's recommended that you approach this with a willingness to experiment, no aversion to reading documentation, and some prior experience with C.

It's quite easy to see what works and what doesn't. As you're finetuning your library list, bear in mind that for the majority of XS builds you may not need the Math library or the C libraries. You may not need the Interface library either. So if there appear to be problems you can always start out with just the runtime library and PerlStub, and look at the linker errors. The Codewarrior Find Library utility is extremely useful when finding out where symbols are defined. Target MacOS in the Codewarrior documentation also discusses which libraries should be used for which projects.

Link order: you may have to fiddle with this. For example, I link the Math libraries first, because for one build the XS was calling C functions with the same name as some Perl functions, which are of course also found in PerlStub. So if you decide to call some other C functions with Perl counterparts, that are in one of the C libraries, expect the possibility of linking order problems. DumpPEF is a very useful MPW tool for diagnosing which library was used to link any particular symbol in your final shared library. And this is when you would use your link map in the Codewarrior IDE.

Math Library: To the best of my recollection the Math library I'm using above is one that I built with the far option. I am also not convinced that it's absolutely necessary - start out by trying the Math libraries that you have available. If there ever is some suspicion that a custom library may solve some problems, all the .mcp library projects are in the Codewarrior distribution. Just open the most relevant one, edit it for your purposes, and build a new library.


Hacking Include Files

The only MacPerl-specific include files you really need for most builds are in :MacPerl_Src:sfio:include and :MacPerl_Src:GUSI:include. With the include paths specified in BuildRules.mk, I found that in order to make a #define properly take, in :MacPerl_Src:sfio:include:stdio.h, namely the first one, I had to double it up. That is, take the initial lines

    #ifndef _SFSTDIO_H  /* protect against multiple #includes */
    #define _SFSTDIO_H  1

and the final line

    #endif /* _SFSTDIO_H */

and duplicate them. This works like a charm. I suggest trying a build, any build, without doing this, and see if you need to.

Note: if you're curious, this technique of using these C preprocessor macros to control header file inclusion is called ``using include guards''. What I found in this case is that I had to double up the include guards.


EXAMPLE FILES

These are general examples. I've picked another for Codewarrior specifically. There are also examples available from the MacPerl Pages under this topic.


Suggested Examples

In your working directory, run h2xs -A -n ExampleA. Change directory to the folder ExampleA. Edit ExampleA.xs to look like the finished product below.

Alternatively, if you want to see whether this process works with real files, like the Mac Toolbox, try one of the extensions in :ext:Mac.

Or, finally, just select an existing CPAN XS distribution. I suggest Data-Locations-5.x. This is not an endorsement - it's a good module to start with because there is no porting requirement.


ExampleA.xs

   #ifdef __cplusplus
   extern "C" {
   #endif
   #include "EXTERN.h"
   #include "perl.h"
   #include "XSUB.h"
   #ifdef __cplusplus
   }
   #endif
   
   #include <math.h>
   
   MODULE = ExampleA    PACKAGE = ExampleA      
   
   void
   quadroots(a, b, c)
         double a;
         double b;
         double c;
      PREINIT:
         double root1;
         double root2;
         double d;
         AV *roots;
      PPCODE:
         if (a==0.0)
            XPUSHs(&sv_undef);
         else
         {
            d = sqrt(b*b - 4*a*c);
            root1 = (-b + d) / 2*a;
            root2 = (-b - d) / 2*a;
            if (GIMME == G_SCALAR)
            {
               roots = newAV();
               av_store(roots, 0, newSVnv(root1));
               av_store(roots, 1, newSVnv(root2));
               XPUSHs(newRV((SV *) roots));
            }
            else
            {
               /* GIMME equals G_ARRAY */
               XPUSHs(newSVnv(root1));
               XPUSHs(newSVnv(root2));
            }
         }


DIAGNOSTICS

MPW Build Hacks
From time to time the various build rules won't quite cut it. One way of handling this is to generate Makefile.mk, edit that as described above, and then run

    dmake -n dynamic -f Makefile.mk > mymake.out

This command is actually being run every time by BuildProgram anyway - you just don't notice. Here we're doing it explicitly. The -n switch tells dmake not to execute, -f identifies the makefile, and dynamic is the target to generate the build rules for, which are then written to file mymake.out.

Edit the file mymake.out. Then run it as a command; literally,

    mymake.out

on the MPW command line.

You may have to do this to add include paths or source files, for example. The current MacPerl ExtUtils::MakeMaker (which is called when you do perl Makefile.PL) only picks up source and header files at the top level of the distribution folder.

Relaxed Pointers
This comes up fairly frequently with CPAN builds. You'll get compiler errors about not being able to convert char * to unsigned char *, or vice versa. Just select the compiler relaxed pointers option.

Dynaloader errors
Everything compiled and linked, you're running a test or otherwise using the module, and Dynaloader error -49 occurs. This means that your installation isn't quite right. (It's a fnfError - file not found). Check file locations (see the INSTALLATION ISSUES section above) and fix manually.

Other Dynaloader errors occasionally occur even if the library apparently was successfully built. The Apple documentation on the Code Fragment Manager has a list of these errors. Normally what's happened is that a symbol was linked from the wrong library. You can find out by looking at the link map. Jigger the link order as required (that is, physically swap lines in a build rules file, or drag libraries around in the Link Order window for your CW IDE project).

Installation
If you're using BuildProgram install_dynamic with Codewarrior MPW, check the :blib:lib: folder to ensure that all files were transferred, and to the proper location, as explained in INSTALLATION ISSUES. This doesn't always happen.

When using installme with a CPAN extension, be aware that blib .pm files are overwritten by the original files in the distribution folder. So make sure you make your changes to all copies, or replace your originals with ported versions prior to running installme. This is only an issue if you had to do some porting (for example, fixing Unixisms in the .pm file).


APPENDIX

Here are some other issues which are relevant to all of the build procedures, but don't fit quite so neatly into the flow of the discussion.


NEWLINES

Arrrgh! Sorry - this is one of my pet peeves. Anyway, be aware that '\n' in MacPerl is CR or \015 or x0D. MPW C compilers recognize '\n' in C code as such by default. The compilers running under the Codewarrior IDE do not. So, with the CW IDE, if your C code contains literal '\n' or '\r' characters, select the Map Newlines to CR option.

Also, if you're using functions from project libraries, and those functions in turn contain literal '\n' or '\r' characters, these libraries should have been compiled with that same option. Look for Codewarrior libraries that have NL as part of the name.


AUTHORS

Arved Sandstrom mailto:Arved_37@chebucto.ns.ca
Overall development. Brickbats to be directed here. In my own defense, let me say that I'm not a CS type, and I'm not even a MacOS type. :-)

Alan Fry mailto:ajf@afco.demon.co.uk
Point-man for Apple MPW MacPerl XS. Alan has concentrated on testing Mac Toolbox builds, to good effect. He is responsible for the polish of the final mac_h2xs_GUI.plx GUI front-end to h2xs. He's also the reason the PDF version of this tutorial looks so good.


BUGS AND FEEDBACK

I normally build MacPerl extensions with Codewarrior MPW, so it's likely that some of my statements regarding use of the IDE are overly restrictive. Some experimentation should lead you to better settings. Please let me know how things go.

Some of my other statements may be misleading, or may be wrong (that is, things work, but not the way I say they do). This is a draft document, and I'd like lots of input. If you're a MacOS + C guru, I'd like to hear from you.

There's room for contributions. I fully expect a substantial revision of this document within a month or two. If you'd like to add to, or modify part of, this tutorial, please contact me. Also, if you manage to build an extension and you had to do something out of the ordinary, consider having your description, or your Codewarrior project, submitted to The MacPerl Pages.

Obligatory Warning: This tutorial describes procedures, which by their very nature, can lead to system crashes of various degrees of severity. In actuality I have never had one happen, except for a really bad one when I ran a Toolbox module in MPW (don't do this at home, kids). But, in any case, be aware that you could over-write all of your recipe files...


CREDITS

Matthias Neeracher for writing all the stuff to make this possible.

Chris Nandor for stupendous MacPerl-related efforts, and frequent words of encouragement.

Vicki Brown for kindly hosting this tutorial and related material on The MacPerl Pages.

Paul Schinder and Larry Moore, for helpful comments and overall Perl enthusiasm. Larry, although he may not realize it, really boosted my enthusiasm for this project by finally entering the game as a CPAN tester for CFM68K, which has provided some very useful feedback.

Miro Jurisic for his help, which led directly to a method for building extensions with SC and ILink.

All the various MacPerl XS enthusiasts.


COPYRIGHT

Copyright (c) 1999 Arved Sandstrom. All rights reserved. In practise, just don't modify this thing and re-distribute the modified copy.