I updated the makefiles and made some changes to compile and test GnuPG
on z/OS R10. This email exchange covers the steps taken and some
patches with the intent of merging these changes into the main branch
for GnuPG and its libraries.
Please excuse the cross-post. Some of the hoops that I jumped through
are relevant to all libraries and I don't want to split them apart. If
need be, we can stick to one mailing list. Pointers to help me better
explain what I am doing as always appreciated.
- Hardware Information:
Architecture z9
z/OS release V1R10
Crypto Express II card with the ICSF software (enable /dev/random in RACF)
- GnuPG and related library versions:
gnupg 1.4.9
libgpg-errors 1.7
libgcrypt 1.4.4
libgpgme 1.1.8
- Back story:
z/OS provides is POSIX compliant environment called Unix System
Services. The platform's native character encoding (aka code page or
code set) is not ASCII, but EBCDIC. The US dialect of EBCDIC is
IBM-1047, but there are dialects for other languages. I believe that
the USS can only operate in IBM-1047, and I have put in a request to
find out if that is a true statement. When using the IBM-1047 EBCDIC
code page the USS has the ability to auto-magically convert between
ASCII (ISO8859-1) and EBCDIC (IBM-1047).
Auto-magic conversion of data streams and files depends on environment
variables and extended file attributes called tags. Required
environment variables are as follows:
_BPXK_AUTOCVT="ON"
_CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
_TAG_REDIR_OUT="txt"
_TAG_REDIR_IN="txt"
_TAG_REDIR_ERR="txt"
You should consider the options for _CEE_RUNOPTS as additions to what
you already have in _CEE_RUNOPTS
Files on disk and file descriptors for a process and process themselves
can be tagged with a character code set ID (CCSID). As of V1R10, all
USS program are compiled as 31bit EBCDIC with no Unicode capabilities.
Breaking this down into a more sensible way:
-- Applications
Applications can have CCSIDs tagged to them defining how they will
internally treat character streams and how they will interact with the
USS. At the moment on the only compilation options are IBM-1047 for
EBCDIC and ISO8859-1 for ASCII. IMHO applications ported to z/OS should
be compiled with the ASCII CCSID to avoid problems such as assuming a-z
is a contiguous block (not true for any form of EBCDIC).
-- Files
Files can be tagged with a CCSID, without a CCSID or as binary. Tagging
a file with a CCSID is meant to tell any opening application the
character encoding inside the file. As of V1R10, this information is
valuable for CCSIDs IBM-1047 (EBCDIC) and ISO8859-1 (ASCII). For the
aforementioned CCSIDs with the
-- File Descriptors
When creating files with file descriptor operations applications must
set file tags correctly to enable conversion. Applications developers
must also decide how to handle untagged files and what to do when file
that they own have been mis-tagged.
-- File Pointers
Using fopen on a file and then fread/fwrite simplifies file tagging and
auto conversion if you have the above environment variables defined.
See the reference link "Steps for setting up Enhanced ASCII" below.
-- Exceptions to the above rules
Files that are tagged with Binary are not subject to auto conversion.
Files tagged with a CCSID but marked as "mixed" text and binary data are
not subject to auto conversion. Files must be tagged as "text" and a
valid CCSID to enable auto conversion.
- Compiling GnuPG et al on z/OS
My team used the XLC compiler from IBM and compiled the libraries and
applications as ASCII (ISO8859-1). We have used a number of MACROs
called Feature Test MACROs to enable specific features and
compatibility. For information on "Feature Test Macros", see
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.bpxbd00/ftms.htm
I tried modifying the configure scripts, but my understanding of
autotools is weak. Where I wanted to build shared libraries, I compiled
archives. I unpacked the archives and converted them to shared libs
manually.
-- libgpg-error
Apply the attached patch libgpg-error-1.7.patch which applies to
libgpg-error 1.7.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--enable-shared --prefix=/usr/local
-- libgcrypt
Apply the attached patch libgcrypt-1.4.4.patch which applies to
libgcrypt 1.4.4.
There is an issue with the xlc configuration that defaults to picking up
include files from '.' (current working directory) and /usr/include
before picking up command line options. Since the z/OS system has two
headers with the same name as headers in libgcrypt, we link them into
the mpi subdirectory to force xlc to use the local headers and not the
system headers in /usr/include. From the source directory root, I did
the following:
cd mpi
ln -s ../src/mpi.h .
ln -s ../src/memory.h .
cd -
cd cipher
ln -s ../src/mpi.h .
cd -
It is possible to create an xlc configuration file to get around this
issue, but that is another beast altogether.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--enable-shared --prefix=/usr/local --with-gpg-error-prefix=/usr/local
-- gpgme
Please use the gpgme-1.2.0 sources which have a fix that is needed for
1.1.8 which I built and tested on z/OS.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--enable-shared --prefix=/usr/local--with-gpg-error-prefix=/usr/local
-- gnupg-1.4.9
Apply the attached patch gnupg-1.4.9.patch which applies to gnupg
1.4.9. I will need help migrating this to the v2 line of gnupg
There is an issue with the xlc configuration that defaults to picking up
include files from '.' (current working directory) and /usr/include
before picking up command line options. Since the z/OS system has two
headers with the same name as headers in libgcrypt, we link them into
the mpi subdirectory to force xlc to use the local headers and not the
system headers in /usr/include. From the source directory root, I did
the following:
cd mpi
ln -s ../include/mpi.h .
ln -s ../include/memory.h .
cd -
It is possible to create an xlc configuration file to get around this
issue, but that is another beast altogether.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--without-pth --without-libassuan --without-ksba --prefix=/usr/local
-- Generating Shared Libs from archives
Since we needed shared libraries, I ripped the object files out of each
library and relinked them into a shared library (DLL on z/OS). The
following is an example of what I did to make gpgme a shared library.
mkdir /tmp/tmp-gpgme
cd /tmp/tmp-gpgme
ar -x /usr/local/lib/libgpgme.a
xlc -qascii -q64 -W l,DLL -o libgpgme.dll *.o /usr/local/lib/libgpg-error.x
cp libgpgme.dll /usr/local/lib
cp libgpgme.x /usr/local/lib
cd -
rm -rf /tmp/tmp-gpgme
- References
Porting to z/OS Unix -
http://www-03.ibm.com/servers/eserver/zseries/zos/unix/bpxa1por.html
Steps for setting up Enhanced ASCII -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.bpxb200/stepas.htm#stepas
z/OS Unix -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.bpx/bpx.htm
z/OS Lanuage Env (include programming guide and run time reference) -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.cee/cee.htm
Information about DLLs -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.cbcpx01/dllsim.htm
- Patches
Are attachments.
_____________
The information contained in this message is proprietary and/or confidential. If you are not the
intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review by
persons other than the intended recipient. Thank you.
_____________
on z/OS R10. This email exchange covers the steps taken and some
patches with the intent of merging these changes into the main branch
for GnuPG and its libraries.
Please excuse the cross-post. Some of the hoops that I jumped through
are relevant to all libraries and I don't want to split them apart. If
need be, we can stick to one mailing list. Pointers to help me better
explain what I am doing as always appreciated.
- Hardware Information:
Architecture z9
z/OS release V1R10
Crypto Express II card with the ICSF software (enable /dev/random in RACF)
- GnuPG and related library versions:
gnupg 1.4.9
libgpg-errors 1.7
libgcrypt 1.4.4
libgpgme 1.1.8
- Back story:
z/OS provides is POSIX compliant environment called Unix System
Services. The platform's native character encoding (aka code page or
code set) is not ASCII, but EBCDIC. The US dialect of EBCDIC is
IBM-1047, but there are dialects for other languages. I believe that
the USS can only operate in IBM-1047, and I have put in a request to
find out if that is a true statement. When using the IBM-1047 EBCDIC
code page the USS has the ability to auto-magically convert between
ASCII (ISO8859-1) and EBCDIC (IBM-1047).
Auto-magic conversion of data streams and files depends on environment
variables and extended file attributes called tags. Required
environment variables are as follows:
_BPXK_AUTOCVT="ON"
_CEE_RUNOPTS="FILETAG(AUTOCVT,AUTOTAG) POSIX(ON)"
_TAG_REDIR_OUT="txt"
_TAG_REDIR_IN="txt"
_TAG_REDIR_ERR="txt"
You should consider the options for _CEE_RUNOPTS as additions to what
you already have in _CEE_RUNOPTS
Files on disk and file descriptors for a process and process themselves
can be tagged with a character code set ID (CCSID). As of V1R10, all
USS program are compiled as 31bit EBCDIC with no Unicode capabilities.
Breaking this down into a more sensible way:
-- Applications
Applications can have CCSIDs tagged to them defining how they will
internally treat character streams and how they will interact with the
USS. At the moment on the only compilation options are IBM-1047 for
EBCDIC and ISO8859-1 for ASCII. IMHO applications ported to z/OS should
be compiled with the ASCII CCSID to avoid problems such as assuming a-z
is a contiguous block (not true for any form of EBCDIC).
-- Files
Files can be tagged with a CCSID, without a CCSID or as binary. Tagging
a file with a CCSID is meant to tell any opening application the
character encoding inside the file. As of V1R10, this information is
valuable for CCSIDs IBM-1047 (EBCDIC) and ISO8859-1 (ASCII). For the
aforementioned CCSIDs with the
-- File Descriptors
When creating files with file descriptor operations applications must
set file tags correctly to enable conversion. Applications developers
must also decide how to handle untagged files and what to do when file
that they own have been mis-tagged.
-- File Pointers
Using fopen on a file and then fread/fwrite simplifies file tagging and
auto conversion if you have the above environment variables defined.
See the reference link "Steps for setting up Enhanced ASCII" below.
-- Exceptions to the above rules
Files that are tagged with Binary are not subject to auto conversion.
Files tagged with a CCSID but marked as "mixed" text and binary data are
not subject to auto conversion. Files must be tagged as "text" and a
valid CCSID to enable auto conversion.
- Compiling GnuPG et al on z/OS
My team used the XLC compiler from IBM and compiled the libraries and
applications as ASCII (ISO8859-1). We have used a number of MACROs
called Feature Test MACROs to enable specific features and
compatibility. For information on "Feature Test Macros", see
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.bpxbd00/ftms.htm
I tried modifying the configure scripts, but my understanding of
autotools is weak. Where I wanted to build shared libraries, I compiled
archives. I unpacked the archives and converted them to shared libs
manually.
-- libgpg-error
Apply the attached patch libgpg-error-1.7.patch which applies to
libgpg-error 1.7.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--enable-shared --prefix=/usr/local
-- libgcrypt
Apply the attached patch libgcrypt-1.4.4.patch which applies to
libgcrypt 1.4.4.
There is an issue with the xlc configuration that defaults to picking up
include files from '.' (current working directory) and /usr/include
before picking up command line options. Since the z/OS system has two
headers with the same name as headers in libgcrypt, we link them into
the mpi subdirectory to force xlc to use the local headers and not the
system headers in /usr/include. From the source directory root, I did
the following:
cd mpi
ln -s ../src/mpi.h .
ln -s ../src/memory.h .
cd -
cd cipher
ln -s ../src/mpi.h .
cd -
It is possible to create an xlc configuration file to get around this
issue, but that is another beast altogether.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--enable-shared --prefix=/usr/local --with-gpg-error-prefix=/usr/local
-- gpgme
Please use the gpgme-1.2.0 sources which have a fix that is needed for
1.1.8 which I built and tested on z/OS.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--enable-shared --prefix=/usr/local--with-gpg-error-prefix=/usr/local
-- gnupg-1.4.9
Apply the attached patch gnupg-1.4.9.patch which applies to gnupg
1.4.9. I will need help migrating this to the v2 line of gnupg
There is an issue with the xlc configuration that defaults to picking up
include files from '.' (current working directory) and /usr/include
before picking up command line options. Since the z/OS system has two
headers with the same name as headers in libgcrypt, we link them into
the mpi subdirectory to force xlc to use the local headers and not the
system headers in /usr/include. From the source directory root, I did
the following:
cd mpi
ln -s ../include/mpi.h .
ln -s ../include/memory.h .
cd -
It is possible to create an xlc configuration file to get around this
issue, but that is another beast altogether.
Configuration options:
./configure CC=xlc CFLAGS="-qchars=signed -qascii -q64 -qlanglvl=extc99
-qexportall -qrent -qnocsect -W l,DLL -D_XOPEN_SOURCE=600
-D_ENHANCED_ASCII_EXT=0xFFFFFFFF -D_IEEEV1_COMPATIBILITY
-D_OPEN_MSGQ_EXT" LD=xlc LDFLAGS="-qascii -q64 -W l,DLL" CXX=xlc++
--without-pth --without-libassuan --without-ksba --prefix=/usr/local
-- Generating Shared Libs from archives
Since we needed shared libraries, I ripped the object files out of each
library and relinked them into a shared library (DLL on z/OS). The
following is an example of what I did to make gpgme a shared library.
mkdir /tmp/tmp-gpgme
cd /tmp/tmp-gpgme
ar -x /usr/local/lib/libgpgme.a
xlc -qascii -q64 -W l,DLL -o libgpgme.dll *.o /usr/local/lib/libgpg-error.x
cp libgpgme.dll /usr/local/lib
cp libgpgme.x /usr/local/lib
cd -
rm -rf /tmp/tmp-gpgme
- References
Porting to z/OS Unix -
http://www-03.ibm.com/servers/eserver/zseries/zos/unix/bpxa1por.html
Steps for setting up Enhanced ASCII -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.bpxb200/stepas.htm#stepas
z/OS Unix -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.bpx/bpx.htm
z/OS Lanuage Env (include programming guide and run time reference) -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.cee/cee.htm
Information about DLLs -
http://publib.boulder.ibm.com/infocenter/zos/v1r10/topic/com.ibm.zos.r10.cbcpx01/dllsim.htm
- Patches
Are attachments.
_____________
The information contained in this message is proprietary and/or confidential. If you are not the
intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review by
persons other than the intended recipient. Thank you.
_____________
Attached Files:
-
gnupg-1.4.9.patch (29.2 KB)
-
libgcrypt-1.4.4.patch (1.55 KB)
-
libgpg-error-1.7.patch (0.81 KB)