Renaud, this is mostly directed to you but others are encouraged to comment
also.
----------------------------------------------------------------------------------------------
Background:
When 2.0.10a came out and most of the patches necessary to run as a 64 bit
application on AMD64 systems had not been included, I decided to do a bit
more research before redoing the patches. One of the interesting pieces of
information I came across was this from one of the articles:
"Developers planning to update code for execution on a 64-bit platform
must deal with several issues besides the obvious change to
64-bit addressing in their source code. For example, the 64-bit
operating systems and compilers use different memory models than
32-bit OSs do. In fact, even Linux and Windows are different.
64-bit Windows uses the LLP64 (or P64) model where a "long" integer
remains 32-bits. All Unix and Linux 64-bit implementations follow
the LP64 model where a "long" integer becomes 64-bits."
The full set of articles/documents I found include the following:
AMD64 Developer Resource Kit:
http://www.amd.com/us-en/Processors/DevelopWithAMD/0,,30_2252_9044,00.html
Presentation on Porting on MS Windows:
http://www.amd.com/us-en/assets/content_type/DownloadableAssets/dwamd_AMD_TechEdEMEA2003_Final.pdf
A general presentation on software porting
http://www.amd.com/us-en/assets/content_type/DownloadableAssets/dwamd_Software_Porting_-_Rich_Brunner.pdf
Another porting guide for Win64:
http://www.devx.com/amd/Article/17783
-----------------------------------------------------------------------------------------------------------
Summary of variable type situation:
x32 = i386, i486, i568, i686, Sparc32, etc. and other 32 bit application
systems
x64 = Athlon64, Opteron, x86_64, Sparc64 and other 64 bit mode application
systems
sizes given in bytes
int: x32=4, x64=4
pointer: x32=4, x64=8
long: x32=4, *nix x64=8, MS Win x64=4
size_t: x32=4, x64=8 (yes, polymorphic on *nix, Linux and Windows)
ssize_t: x32=4, x64=8 (yes, polymorphic on *nix, Linux and Windows)
long long: x32=8, x64=8
Microsoft also defines some additional polymorphic types dealing with pointers
but these are not defined on Linux/glibc and I doubt that they are defines on
any *nix.
------------------------------------------------------------------------------------------------
Bugs and submitted patches:
1. At least one of the updates [sizeof(ifmap)] cases problems on Solaris and
possibly other systems too. This will need more work to make things work
properly on all systems.
2. While the rlimit patch I submitted works OK, the setting of memory limits
needs to be a runtime option in /etc/nessus/nessusd.conf.
3. Many/most of the lack of prototypes from the SUSE and other patches have
been incorporated in 2.0.10a. I have found a problem on the amd64 with
memmem() where the prototype from <string.h> still causes cast problems but a
locally defined prototype (using the definition in the man page) does not.
This needs more investigation by me.
4. You have rejected most of the porting/cast fixes developed by SUSE and
this was the CORRECT thing to do. These patches are just plain wrong in what
they do.
a) The inclusion of <inttypes.h> is not portable.
b) The strange format strings are even worse. In the above documents,
Microsoft recommends using "%p" to print pointers and this makes sense to me
for debugging related statements.
c) The use of the intptr_t cast to "fix" the cast warnings is just wrong.
This resulted in no warnings but pointers were truncated to 4 bytes or zero
expanded from 4 byte "int" storage to an 8 byte pointer. While this works OK
as long as a virtual storage address was below 4GB, it will only be a
headache for someone later AND is simply poor coding practice. As the
porting recommendations referenced above say -- don't just port to 64 bits
but make the software architecture independent.
-------------------------------------------------------------------------------------------------
So, where to go from here? I don't want to waste a lot of my time (and your
time) developing patches which are rejected and/or cause problems on other
systems (I run Linux). However, I also do not want to have a whole bunch of
Linux-only patches which someone will need to maintain -- I want the work to
be incorporated into the base.
Here are some of the options/ideas I have come up with (additions/alternatives
would be appreciated):
1. For debugging related output of pointers use "%p". If it is not clear that
the output is a pointer (or if the value varies) then use "%ld". This should
work for all cases I can think of. For Microsoft Windows and nessus, I
believe most of the formatting is debugging related and pointers but this
needs a bit more research.
2. For pointer related variables [and function parameters and returned values]
which will/may hold a pointer, we need something that will size to 4 bytes on
IA32 and to 8 bytes on the amd64 and other 64 bit architectures. It would be
very nice to continue to size variables which truly only need 32 bits as 4
byte variables. The obvious (to me) solutions is to use long and unsigned
long (and size_t and ssize_t in nessus for MS Windows [MS recommended
solution]).
a) We can globally change all "int" to "long" (or "size_t" for nessus).
b) Alternatively, we can define our own typedef and use that for all variables
which could possibly hold a pointer.
c) We can review/recode every structure to adequately provide for the
possibility of storing 8 byte pointers (using unions). Then we could
review/change every function so that it specifies and handles the possible
pointer value properly. While some this is likely a good idea, it seems way
too much of an effort to be practical. This would be almost like rebuilding
nessus from the ground up.
(a) or (b) are possibly the easiest approaches although it would still take a
bit of work. To me, I cannot see a penalty that would be paid on a x32
system. My preference is (b).
Any other ideas?? Any problems someone can raise with my proposals??
While I regret that pointers were stored into int variables, it was done and
it worked (at least on 32 bit systems). Now lets deal with it to get things
to work properly.
Renaud, you "own" this code. What do you want to do?
--
Gene Czarcinski
also.
----------------------------------------------------------------------------------------------
Background:
When 2.0.10a came out and most of the patches necessary to run as a 64 bit
application on AMD64 systems had not been included, I decided to do a bit
more research before redoing the patches. One of the interesting pieces of
information I came across was this from one of the articles:
"Developers planning to update code for execution on a 64-bit platform
must deal with several issues besides the obvious change to
64-bit addressing in their source code. For example, the 64-bit
operating systems and compilers use different memory models than
32-bit OSs do. In fact, even Linux and Windows are different.
64-bit Windows uses the LLP64 (or P64) model where a "long" integer
remains 32-bits. All Unix and Linux 64-bit implementations follow
the LP64 model where a "long" integer becomes 64-bits."
The full set of articles/documents I found include the following:
AMD64 Developer Resource Kit:
http://www.amd.com/us-en/Processors/DevelopWithAMD/0,,30_2252_9044,00.html
Presentation on Porting on MS Windows:
http://www.amd.com/us-en/assets/content_type/DownloadableAssets/dwamd_AMD_TechEdEMEA2003_Final.pdf
A general presentation on software porting
http://www.amd.com/us-en/assets/content_type/DownloadableAssets/dwamd_Software_Porting_-_Rich_Brunner.pdf
Another porting guide for Win64:
http://www.devx.com/amd/Article/17783
-----------------------------------------------------------------------------------------------------------
Summary of variable type situation:
x32 = i386, i486, i568, i686, Sparc32, etc. and other 32 bit application
systems
x64 = Athlon64, Opteron, x86_64, Sparc64 and other 64 bit mode application
systems
sizes given in bytes
int: x32=4, x64=4
pointer: x32=4, x64=8
long: x32=4, *nix x64=8, MS Win x64=4
size_t: x32=4, x64=8 (yes, polymorphic on *nix, Linux and Windows)
ssize_t: x32=4, x64=8 (yes, polymorphic on *nix, Linux and Windows)
long long: x32=8, x64=8
Microsoft also defines some additional polymorphic types dealing with pointers
but these are not defined on Linux/glibc and I doubt that they are defines on
any *nix.
------------------------------------------------------------------------------------------------
Bugs and submitted patches:
1. At least one of the updates [sizeof(ifmap)] cases problems on Solaris and
possibly other systems too. This will need more work to make things work
properly on all systems.
2. While the rlimit patch I submitted works OK, the setting of memory limits
needs to be a runtime option in /etc/nessus/nessusd.conf.
3. Many/most of the lack of prototypes from the SUSE and other patches have
been incorporated in 2.0.10a. I have found a problem on the amd64 with
memmem() where the prototype from <string.h> still causes cast problems but a
locally defined prototype (using the definition in the man page) does not.
This needs more investigation by me.
4. You have rejected most of the porting/cast fixes developed by SUSE and
this was the CORRECT thing to do. These patches are just plain wrong in what
they do.
a) The inclusion of <inttypes.h> is not portable.
b) The strange format strings are even worse. In the above documents,
Microsoft recommends using "%p" to print pointers and this makes sense to me
for debugging related statements.
c) The use of the intptr_t cast to "fix" the cast warnings is just wrong.
This resulted in no warnings but pointers were truncated to 4 bytes or zero
expanded from 4 byte "int" storage to an 8 byte pointer. While this works OK
as long as a virtual storage address was below 4GB, it will only be a
headache for someone later AND is simply poor coding practice. As the
porting recommendations referenced above say -- don't just port to 64 bits
but make the software architecture independent.
-------------------------------------------------------------------------------------------------
So, where to go from here? I don't want to waste a lot of my time (and your
time) developing patches which are rejected and/or cause problems on other
systems (I run Linux). However, I also do not want to have a whole bunch of
Linux-only patches which someone will need to maintain -- I want the work to
be incorporated into the base.
Here are some of the options/ideas I have come up with (additions/alternatives
would be appreciated):
1. For debugging related output of pointers use "%p". If it is not clear that
the output is a pointer (or if the value varies) then use "%ld". This should
work for all cases I can think of. For Microsoft Windows and nessus, I
believe most of the formatting is debugging related and pointers but this
needs a bit more research.
2. For pointer related variables [and function parameters and returned values]
which will/may hold a pointer, we need something that will size to 4 bytes on
IA32 and to 8 bytes on the amd64 and other 64 bit architectures. It would be
very nice to continue to size variables which truly only need 32 bits as 4
byte variables. The obvious (to me) solutions is to use long and unsigned
long (and size_t and ssize_t in nessus for MS Windows [MS recommended
solution]).
a) We can globally change all "int" to "long" (or "size_t" for nessus).
b) Alternatively, we can define our own typedef and use that for all variables
which could possibly hold a pointer.
c) We can review/recode every structure to adequately provide for the
possibility of storing 8 byte pointers (using unions). Then we could
review/change every function so that it specifies and handles the possible
pointer value properly. While some this is likely a good idea, it seems way
too much of an effort to be practical. This would be almost like rebuilding
nessus from the ground up.
(a) or (b) are possibly the easiest approaches although it would still take a
bit of work. To me, I cannot see a penalty that would be paid on a x32
system. My preference is (b).
Any other ideas?? Any problems someone can raise with my proposals??
While I regret that pointers were stored into int variables, it was done and
it worked (at least on 32 bit systems). Now lets deal with it to get things
to work properly.
Renaud, you "own" this code. What do you want to do?
--
Gene Czarcinski