Using Gambit with External Libraries

From Gambit wiki

(Difference between revisions)
Jump to: navigation, search
(New page: == Using gsc to compile and link a dynamically loadable object file that uses external libraries == Here is an example of building a dynamically loadable Gambit object file that uses [htt...)
m (Accessing Scheme vectors within a C function)
 
(23 intermediate revisions not shown)
Line 1: Line 1:
 +
== The return strategy required by Gambit apps on Scheme->C->Scheme calls ==
 +
How Gambit integrates the Scheme heap with the C stack is described in the section "19.7 Continuations and the C-interface" in the manual, and also in section 3 of the paper [http://www.iro.umontreal.ca/~feeley/papers/FeeleySW00.pdf A Portable Implementation of First-Class Continuations for Unrestricted Interoperaibility with C in a Multithreaded Scheme]. Here is a clarification though, which was detailed 28 march 2011 on the mailing list:
 +
 +
If you have several C stack frames (produced by a Scheme->C call which made a C->Scheme call that made a Scheme->C call in turn) at the same time, you must return them in the same sequence as you'd have needed to do ordinarily in C, i.e. from the last to the first in sequence.
 +
 +
I.e., if you have the Scheme procedures A, C, E and G, and the C procedures b, d and f, and they invoke each other A -> b -> C -> d -> E -> f -> G, then you must ensure that G will return to f, f to E, E to d, d to C, C to b and b to A.
 +
 +
If you return them in another order - i.e. for example G to d etc. - there will be a runtime error, which terminates the application silently.
 +
 +
Note that anytime during a program's execution in the Scheme world, Gambit's thread multitasker may switch the running thread. If several threads do Scheme->C->Scheme calls at the same time, then in the ordinary case, thread switching may happen such that the C stack is rewinded invalidly (i.e. in another order than as described above), which at some point (not necessarily on the first invalid return) will cause the abovementioned runtime error. You can fix this by
 +
* keeping all Scheme->C->Scheme calls in your app to one thread in total,
 +
* by rewriting your code to do what you wanted to achieve through making a C->Scheme call some other way instead (for instance by using advanced C programming techniques that are beyond the scope of this document),
 +
* or by going with the "Ensuring singlethreaded behavior" described below.
 +
 +
(Advanced note: Actually Gambit allows you to skip returning procedures, i.e. A -> b -> C -> d -> E, and then E returns directly to b, works. On the call to b, d's C stack frame is rewinded though, and returning to d would cause the abovementioned runtime error. This works because Gambit on the call to b makes a longjump that simply disposes of d's stack frame. This strategy could cause stack memory leaks though.
 +
If you by any reason explore how use of this side of the FFI can be made use of, please document it here and on the mailing list.)
 +
 +
== Ensuring singlethreaded behaviour ==
 +
In certain situations, it's vital to ensure a single thread of execution.
 +
 +
One way may be to create one thread to which you send closures containing code to be executed, and which returns the responses through a mailbox mechanism, there's an example implementation in the Gambit manual.
 +
 +
Ways to get Gambit execute completely single-threaded is:
 +
 +
* Use (thread-quantum-set! (current-thread) +inf.0)
 +
 +
* Use (##disable-interrupts) and (##enable-interrupts) in Scheme or ___EXT(___disable_interrupts)() and ___EXT(___enable_interrupts)() from C.
 +
 +
Please note that Gambit's I/O system makes use of the scheduler, and threading routines do this also, so don't do read, write, thread-sleep!, thread-yield! etc. in code you intended to execute single-threaded.
 +
 +
== Export and import C symbols ==
 +
Gambit's gambit.h provides helper macros for exporting functions and variables. They are  ___EXPORT_FUNC(type,name) and ___EXPORT_DATA(type,name), and are used like ___EXPORT_FUNC(int,five) () { return 5; } . Grep lib/*.c of the Gambit sources for EXP_FUNC and EXP_DATA to see examples.
 +
 +
On Windows, exporting and importing functions and variables from C code may be particularly tricky. Check out the Microsoft-provided __declspec(dllexport) and __declspec(dllimport).
 +
== Using gsc to compile and link a dynamically loadable object file that uses external libraries ==
== Using gsc to compile and link a dynamically loadable object file that uses external libraries ==
Line 53: Line 88:
We need to pass special options to gsc to compile this file, namely
We need to pass special options to gsc to compile this file, namely
<pre>
<pre>
-
gsc -cc-options "-I/export/users/lucier/local/fftw-2.1.5/include" -ld-options "-L/export/users/lucier/local/fftw-2.1.5/lib/ -Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/ -lfftw" fftbasic.scm
+
gsc -cc-options "-I/export/users/lucier/local/fftw-2.1.5/include" \
 +
    -ld-options "-L/export/users/lucier/local/fftw-2.1.5/lib/ -Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/ -lfftw" fftbasic.scm
</pre>
</pre>
The first option (<tt>-I/export/users/lucier/local/fftw-2.1.5/include</tt>) tells gcc where to find the header file <tt>fftw.h</tt> at compile time.  The second option (<tt>-L/export/users/lucier/local/fftw-2.1.5/lib/</tt>) tells the linker where to find the FFTW library (<tt>-lfftw</tt>) at link time (i.e., when building the file <tt>fftwbasic.o1</tt> from <tt>fftwbasic.o</tt>), and the third option (<tt>-Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/</tt>) tells the dynamic loader <tt>ldd</tt> where to find the FFTW library when <tt>fftwbasic.o1</tt> is loaded into gsc.
The first option (<tt>-I/export/users/lucier/local/fftw-2.1.5/include</tt>) tells gcc where to find the header file <tt>fftw.h</tt> at compile time.  The second option (<tt>-L/export/users/lucier/local/fftw-2.1.5/lib/</tt>) tells the linker where to find the FFTW library (<tt>-lfftw</tt>) at link time (i.e., when building the file <tt>fftwbasic.o1</tt> from <tt>fftwbasic.o</tt>), and the third option (<tt>-Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/</tt>) tells the dynamic loader <tt>ldd</tt> where to find the FFTW library when <tt>fftwbasic.o1</tt> is loaded into gsc.
 +
 +
<b>Aside</b>: Note that if the headers and libraries are in a standard place known to gcc, and the location of the shared library is already in the path of the dynamic loader, then these options may not be necessary.  In many GNU/Linux systems, for examples, nearly all packages are installed in <tt>/usr/{bin,include,lib}</tt>, and you may not need to pass these special options to gsc.
Then we can do
Then we can do
Line 77: Line 115:
         /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
         /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
</pre>
</pre>
 +
 +
Finally, recall from the the [http://www.iro.umontreal.ca/~gambit/doc/gambit-c.html#SEC21 Gambit manual] that anything you can do with gsc on the command line you can do with one of the gsc-specific scheme procedures <tt>compile-file</tt>, <tt>compile-file-to-c</tt>, <tt>link-incremental</tt>, or <tt>link-flat</tt>.  Thus, one could build <tt>fftbasic.o1</tt> by
 +
<pre>
 +
euler-352% gsc
 +
Gambit v4.2.8
 +
 +
> (compile-file "fftbasic.scm" cc-options: "-I/export/users/lucier/local/fftw-2.1.5/include"
 +
ld-options: "-L/export/users/lucier/local/fftw-2.1.5/lib/ -Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/ -lfftw")
 +
#t
 +
> (load "fftbasic")
 +
"/export/users/lucier/programs/gambc-v4_2_8/test-load-options/fftbasic.o1"
 +
> fftwc
 +
#<procedure #2 fftwc>
 +
</pre>
 +
 +
 +
== Accessing Scheme vectors within a C function ==
 +
 +
Example. Get the pointer to the beginning of a u8vector Scheme object:
 +
 +
<pre>
 +
(define ffi-with-scheme-vectors
 +
  (c-lambda (scheme-object int) ; scheme-object : the vector , int : the vector size
 +
            void
 +
            "
 +
//void *u8vectorptr = ___CAST(void*,&___FETCH_U8(___BODY(___arg1),___INT(0)));
 +
//void *u8vectorptr = ___CAST(void*,&___FETCH_U8(___arg1,0));
 +
//void *u8vectorptr = ___CAST(void*,___BODY(___arg1));
 +
//void *u8vectorptr = ___CAST(___U8*,___BODY_AS(___arg1,___tSUBTYPED));
 +
 +
// Of course, you can cast directly to uchar* if you plan to work with that
 +
unsigned char *u8vectorptr = ___CAST(___U8*,___BODY_AS(___arg1,___tSUBTYPED));
 +
 +
/* Then here do your work with *u8vectorptr, you have its size as the ___arg2 argument */
 +
            "))
 +
</pre>
 +
 +
(Note: There are macros also to get vector length. Note that some of the vector-related deal with vector size in bytes not elements, and that they may return the length as a Gambit fixnum so you need to run it through the ___INT macro to get it in C int format.)
 +
 +
Look for examples in "gambit.h"
 +
 +
Caveat: the C compiler does not know that the GC might move objects, so the C code must be written to avoid calling the GC either directly or indirectly. Remember that the pointer is only to be kept until the next return to Scheme.
 +
 +
== Practices in FFI development ==
 +
(There are a couple of posts from September 2008 in the mailing list archive on this subject. Someone please cut and paste them over here.)
 +
  https://mercure.iro.umontreal.ca/pipermail/gambit-list/2008-September/002572.html
 +
 +
[[Category: FFI]]

Latest revision as of 00:50, 3 February 2013

Contents

The return strategy required by Gambit apps on Scheme->C->Scheme calls

How Gambit integrates the Scheme heap with the C stack is described in the section "19.7 Continuations and the C-interface" in the manual, and also in section 3 of the paper A Portable Implementation of First-Class Continuations for Unrestricted Interoperaibility with C in a Multithreaded Scheme. Here is a clarification though, which was detailed 28 march 2011 on the mailing list:

If you have several C stack frames (produced by a Scheme->C call which made a C->Scheme call that made a Scheme->C call in turn) at the same time, you must return them in the same sequence as you'd have needed to do ordinarily in C, i.e. from the last to the first in sequence.

I.e., if you have the Scheme procedures A, C, E and G, and the C procedures b, d and f, and they invoke each other A -> b -> C -> d -> E -> f -> G, then you must ensure that G will return to f, f to E, E to d, d to C, C to b and b to A.

If you return them in another order - i.e. for example G to d etc. - there will be a runtime error, which terminates the application silently.

Note that anytime during a program's execution in the Scheme world, Gambit's thread multitasker may switch the running thread. If several threads do Scheme->C->Scheme calls at the same time, then in the ordinary case, thread switching may happen such that the C stack is rewinded invalidly (i.e. in another order than as described above), which at some point (not necessarily on the first invalid return) will cause the abovementioned runtime error. You can fix this by

  • keeping all Scheme->C->Scheme calls in your app to one thread in total,
  • by rewriting your code to do what you wanted to achieve through making a C->Scheme call some other way instead (for instance by using advanced C programming techniques that are beyond the scope of this document),
  • or by going with the "Ensuring singlethreaded behavior" described below.

(Advanced note: Actually Gambit allows you to skip returning procedures, i.e. A -> b -> C -> d -> E, and then E returns directly to b, works. On the call to b, d's C stack frame is rewinded though, and returning to d would cause the abovementioned runtime error. This works because Gambit on the call to b makes a longjump that simply disposes of d's stack frame. This strategy could cause stack memory leaks though. If you by any reason explore how use of this side of the FFI can be made use of, please document it here and on the mailing list.)

Ensuring singlethreaded behaviour

In certain situations, it's vital to ensure a single thread of execution.

One way may be to create one thread to which you send closures containing code to be executed, and which returns the responses through a mailbox mechanism, there's an example implementation in the Gambit manual.

Ways to get Gambit execute completely single-threaded is:

  • Use (thread-quantum-set! (current-thread) +inf.0)
  • Use (##disable-interrupts) and (##enable-interrupts) in Scheme or ___EXT(___disable_interrupts)() and ___EXT(___enable_interrupts)() from C.

Please note that Gambit's I/O system makes use of the scheduler, and threading routines do this also, so don't do read, write, thread-sleep!, thread-yield! etc. in code you intended to execute single-threaded.

Export and import C symbols

Gambit's gambit.h provides helper macros for exporting functions and variables. They are ___EXPORT_FUNC(type,name) and ___EXPORT_DATA(type,name), and are used like ___EXPORT_FUNC(int,five) () { return 5; } . Grep lib/*.c of the Gambit sources for EXP_FUNC and EXP_DATA to see examples.

On Windows, exporting and importing functions and variables from C code may be particularly tricky. Check out the Microsoft-provided __declspec(dllexport) and __declspec(dllimport).

Using gsc to compile and link a dynamically loadable object file that uses external libraries

Here is an example of building a dynamically loadable Gambit object file that uses FFTW. This example is on Red Hat Enterprise Linux 4.2 on x86-64.

The program uses the FFTW version 2 API, so we downloaded fftw-2.1.5.tar.gz, untarred it and configured it with

./configure --enable-shared --prefix=/export/users/lucier/local/fftw-2.1.5

You need the --enable-shared option because shared Gambit modules must be linked to shared external libraries. I set the --prefix to install the final FFTW libraries and header files in my home directory.

The file fftbasics.scm provides the basic interface between the Scheme code and FFTW; it is as follows:

(c-declare
"
#include \"fftw.h\"

fftwnd_plan p;

")

(define fftw2d_create_plan_backward
  (c-lambda ()
            void
            "p = fftw2d_create_plan(64,
                                    64,
                                    FFTW_BACKWARD,
                                    FFTW_ESTIMATE | FFTW_IN_PLACE);
            "))

(define fftw2d_create_plan_forward
  (c-lambda ()
            void
            "p = fftw2d_create_plan(64,
                                    64,
                                    FFTW_FORWARD,
                                    FFTW_ESTIMATE | FFTW_IN_PLACE);
            "))

;;; Both forward and backward ffts, depends on which way the plan was created.

(define fftwc
  (c-lambda (scheme-object)
            void
            "
int j; double *fp = (double *)((___WORD)___BODY_AS(___arg1,___tSUBTYPED));
  fftwnd_one(p,
             (fftw_complex *)(fp),
             NULL);
  for (j = 0; j < 64 * 64 * 2; j++)
    fp[j] *= .015625;
"))

We need to pass special options to gsc to compile this file, namely

gsc -cc-options "-I/export/users/lucier/local/fftw-2.1.5/include" \
    -ld-options "-L/export/users/lucier/local/fftw-2.1.5/lib/ -Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/ -lfftw" fftbasic.scm

The first option (-I/export/users/lucier/local/fftw-2.1.5/include) tells gcc where to find the header file fftw.h at compile time. The second option (-L/export/users/lucier/local/fftw-2.1.5/lib/) tells the linker where to find the FFTW library (-lfftw) at link time (i.e., when building the file fftwbasic.o1 from fftwbasic.o), and the third option (-Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/) tells the dynamic loader ldd where to find the FFTW library when fftwbasic.o1 is loaded into gsc.

Aside: Note that if the headers and libraries are in a standard place known to gcc, and the location of the shared library is already in the path of the dynamic loader, then these options may not be necessary. In many GNU/Linux systems, for examples, nearly all packages are installed in /usr/{bin,include,lib}, and you may not need to pass these special options to gsc.

Then we can do

euler-316% gsc
Gambit v4.2.8

> (load "fftbasic")
"/export/users/lucier/programs/gambc-v4_2_8/test-load-options/fftbasic.o1"
> fftwc
#<procedure #2 fftwc>
>
*** EOF again to exit

We can check that fftbasic.o1 links to the right libraries:

euler-317% ldd fftbasic.o1
        libfftw.so.2 => /export/users/lucier/local/fftw-2.1.5/lib/libfftw.so.2 (0x0000002a9565a000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a957aa000)
        libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a959df000)
        /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)

Finally, recall from the the Gambit manual that anything you can do with gsc on the command line you can do with one of the gsc-specific scheme procedures compile-file, compile-file-to-c, link-incremental, or link-flat. Thus, one could build fftbasic.o1 by

euler-352% gsc
Gambit v4.2.8

> (compile-file "fftbasic.scm" cc-options: "-I/export/users/lucier/local/fftw-2.1.5/include"
 ld-options: "-L/export/users/lucier/local/fftw-2.1.5/lib/ -Wl,-rpath,/export/users/lucier/local/fftw-2.1.5/lib/ -lfftw")
#t
> (load "fftbasic")
"/export/users/lucier/programs/gambc-v4_2_8/test-load-options/fftbasic.o1"
> fftwc
#<procedure #2 fftwc>


Accessing Scheme vectors within a C function

Example. Get the pointer to the beginning of a u8vector Scheme object:

(define ffi-with-scheme-vectors
  (c-lambda (scheme-object int) ; scheme-object : the vector , int : the vector size
            void
            "
//void *u8vectorptr = ___CAST(void*,&___FETCH_U8(___BODY(___arg1),___INT(0)));
//void *u8vectorptr = ___CAST(void*,&___FETCH_U8(___arg1,0));
//void *u8vectorptr = ___CAST(void*,___BODY(___arg1));
//void *u8vectorptr = ___CAST(___U8*,___BODY_AS(___arg1,___tSUBTYPED));

// Of course, you can cast directly to uchar* if you plan to work with that
unsigned char *u8vectorptr = ___CAST(___U8*,___BODY_AS(___arg1,___tSUBTYPED));

/* Then here do your work with *u8vectorptr, you have its size as the ___arg2 argument */
            "))

(Note: There are macros also to get vector length. Note that some of the vector-related deal with vector size in bytes not elements, and that they may return the length as a Gambit fixnum so you need to run it through the ___INT macro to get it in C int format.)

Look for examples in "gambit.h"

Caveat: the C compiler does not know that the GC might move objects, so the C code must be written to avoid calling the GC either directly or indirectly. Remember that the pointer is only to be kept until the next return to Scheme.

Practices in FFI development

(There are a couple of posts from September 2008 in the mailing list archive on this subject. Someone please cut and paste them over here.)

  https://mercure.iro.umontreal.ca/pipermail/gambit-list/2008-September/002572.html
Personal tools
documentation