Mailing List Archive

[Bug 139] Dynamically loadable lookup modules
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139


holmgren@lysator.liu.se changed:

What |Removed |Added
----------------------------------------------------------------------------
URL| |http://www.exim.org/eximwiki
| |/NigelMetheringham/Dynamical
| |lyLoadedLookups




------- Comment #1 from holmgren@lysator.liu.se 2007-02-01 17:58 -------
The mentioned patch can be found at
http://www.exim.org/pipermail/exim-users/Week-of-Mon-20000110/016177.html, but
it's quite outdated.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #2 from bugs@timj.co.uk 2007-02-01 18:46 -------
A newer patch (against 4.33) is at:

http://perlgolf.de/exim-4.33-dynamic-load.patch

also note the discussion about this wrt Fedora packages over at:

http://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=199256

and my lengthy message showing the progress I had made at:

http://www.exim.org/mail-archives/exim-dev/2005-December/msg00027.html

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #3 from johannes@sipsolutions.net 2007-02-02 14:26 -------
That patch can't compile, it's missing the new file lookupapi.h

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #4 from bugs@timj.co.uk 2007-02-03 18:14 -------
(In reply to comment #3)
> That patch can't compile, it's missing the new file lookupapi.h

Which patch, specifically? (There are at least 3 separate patches discussed in
this bug)

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #5 from johannes@sipsolutions.net 2007-02-03 23:57 -------
Your last one. But I fixed it up, attaching in a minute, it works for me now.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #6 from johannes@sipsolutions.net 2007-02-04 00:12 -------
Created an attachment (id=56)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=56&action=view)
patch to make dynamically loadable modules possible

This patch makes dynamically loadable lookup modules possible.

Based on Tim's patch, heavily modified, and I had to add lots of
makefile foo...

The one thing I haven't quite sorted out yet is how to install
the modules. I started it, but LOOKUP_MODULE_DIR doesn't seem
right, it should have some prefix like LIBDIR or BINDIR or
something... Someone else can probably figure that out much
much faster than me.

So. Known bugs:
* module install not done
* module name list that is built during loading
is never freed (see addlookupmodule)
* the library ending is hard-coded to .so

Pros:
* well, loadable module support :)
* no more src/lookups/*.h files (except lf_functions.h)
* all lookup functions added dynamically, so can all be static now

Cons:
* lf_quote.c and lf_check_file.c are always linked in

Caveats:
* LOOKUP_XXX=2 was previously a valid setting for compiling it in
* the "Lookups:" printout line changed to
"Lookups (built-in):"

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139


johannes@sipsolutions.net changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #56 is|0 |1
obsolete| |




------- Comment #7 from johannes@sipsolutions.net 2007-02-04 00:36 -------
Created an attachment (id=57)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=57&action=view)
patch to make dynamically loadable modules possible

changes over the previous version:
* missing entry in config.h.defaults
* compilation fix in drtables.c
* LOOKUP_*_LIBS and LOOKUP_*_INCLUDES so that e.g. the mysql library
can be linked only to the mysql.so lookup module.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139


johannes@sipsolutions.net changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #57 is|0 |1
obsolete| |




------- Comment #8 from johannes@sipsolutions.net 2007-02-04 00:41 -------
Created an attachment (id=58)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=58&action=view)
patch to make dynamically loadable modules possible

Ahhrrg. I forgot one little thing in src/EDITME, you don't just need
-export-dynamic but also -rdynamic.

Sorry about the spamming.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #9 from johannes@sipsolutions.net 2007-02-04 01:21 -------
Created an attachment (id=59)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=59&action=view)
fix cdb lookup compilation

Needs to be applied after the other patch to fix cdb compilation.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139





------- Comment #10 from johannes@sipsolutions.net 2007-02-04 01:22 -------
Created an attachment (id=60)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=60&action=view)
fix LDAP

LDAP cannot be compiled as a module due to eldapauth_find, so disallow it. To
be applied after the initial patch.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139


johannes@sipsolutions.net changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #58 is|0 |1
obsolete| |
Attachment #59 is|0 |1
obsolete| |
Attachment #60 is|0 |1
obsolete| |




------- Comment #11 from johannes@sipsolutions.net 2007-02-04 14:09 -------
Created an attachment (id=61)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=61&action=view)
patch to make dynamically loadable modules possible

New version again...
* fold in the two other patches
* fix the two memory leaks introduced by previous versions of the patch:
- lookupmodulestr structs created during module discovery
- RE structure that is created for matching *.so

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
[Bug 139] Dynamically loadable lookup modules [ In reply to ]
------- You are receiving this mail because: -------
You are the QA contact for the bug, or are watching the QA contact.

http://www.exim.org/bugzilla/show_bug.cgi?id=139


johannes@sipsolutions.net changed:

What |Removed |Added
----------------------------------------------------------------------------
Attachment #61 is|0 |1
obsolete| |




------- Comment #12 from johannes@sipsolutions.net 2007-02-04 14:29 -------
Created an attachment (id=62)
--> (http://www.exim.org/bugzilla/attachment.cgi?id=62&action=view)
patch to make dynamically loadable modules possible

Changes:
* SPF cannot be built as a module, it's not just a lookup
* default to no LOOKUP_MODULE_DIR and clarify that it needs
to be set so the exim binary will load modules at all.

--
Configure bugmail: http://www.exim.org/bugzilla/userprefs.cgi?tab=email

--
## List details at http://www.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##
Re: [Bug 139] Dynamically loadable lookup modules [ In reply to ]
On Wed, 2008-05-14 at 14:57 +0100, Tony Finch wrote:
> --- Comment #20 from Tony Finch <dot@dotat.at> 2008-05-14 14:57:21 ---
> Some comments based on a fairly cursory read-through of the patch...

Thanks for the review.

> I'm not sure that configuring the available lookups by installing files in
> a library directory is very nice. I can see it makes sense from a Linux
> distribution point of view, but when building from source or creating a
> unified package it makes sense to install all the lookups then load only
> the necessary ones at run time. I'm not sure if this would lead to a
> chicken/egg problem.

Hm, I haven't really given much thought to use cases _other_ than that
of software distributors who want to make it _possible_, but not
mandatory, to use certain lookups in their prebuilt version.

When you're building from source, why would you ever build modules
dynamically? You can just turn off the ones you don't want (or don't
have the necessary development environment for).

When you say "creating a unified package", you mean a package such as
the existing distro packages, with all the lookups present -- except
that they're as modules, not built-in? That kind of defeats the object
of having them modular, surely?

Certainly, it would be possible to set stuff up so that you have to list
the modules you want loaded, rather than having them get loaded
automatically. I don't really understand the use case where that would
be preferable, though. The reason I _don't_ like it is because it means
you have to edit your config when you (or your distro) switch to
building with dynamically-loaded modules.

I was thinking of moving the contents of the existing 'exim' package to
an 'exim-core' package, and having an empty 'exim' package which pulls
in that _and_ the lookup modules which we used to build statically. That
way, it would have no effect on upgrades. People can still remove the
'exim', exim-pgsql' and 'exim-mysql' packages if they want to save
space, but configurations don't break by default.

> drtables.c:
>
> The #include <dlfcn.h> should be removed for portability to
> systems without that header. Instead add defined(LOOKUP_MODULE_DIR) to
> the #if on line 463 of exim.h which guards another #include <dlfcn.h>.

Fixed (in {http,git}://git.infradead.org/users/dwmw2/exim-dynlookup.git)

> lookupapi.h:
>
> This file is missing the copyright and end-of-file rubric.

University of Cambridge 1995-2007?

I suspect that the end-of-file rubric might be something we could start
to dispense with. (One of these days I might even be brave enough to
suggest running the whole thing through indent, too. It makes my brain
hurt :)

> The LMM1 comment should be less cryptic.

Can you be a little more specific about what you want it to say?
0x4c4d4d41 is "LMM1"... that seems simple enough. Maybe it wants a
comment saying how it's used, but that's a different issue.

> lookups/Makefile:
>
> This file is gmake-only which is a nasty regression. Exim's build system
> has stuff in the scripts directory that constructs the actual Makefile
> based on EDITME. I would greatly prefer it if that mechanism were adapted
> to support the new functionality so that we don't have to add another
> build dependency.
>
> The .c.so rule is non-portable. The shared-library command line options
> need to be made into an EDITME option.

Will play with that; thanks. Which bits are gmake-only?

--
dwmw2


--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##