[PATCH] Add plugin interface to LD [4/4] Add files, libs and dir paths.

[Dave Korn <dave dot korn dot cygwin at gmail dot com>]

  This is the final functionality needed to start supporting the GCC LTO plugin.

- Adding new files/libs/search dirs are fairly trivial callouts to existing
functions in ld, so hopefully the added files should behave just like real
ones, with consistent lang_statements and all.

- It makes use of the bfd linker's multiple-definitions callback to spot when
a freshly-added object file contains a real symbol that was initially supplied
from an IR file, and replaces the old definition with the new one.  This
should correctly cut the dummy bfd right out of the final link, I hope -
unless someone knows some reason why whipping the rug out from under BFD's
feet like this wouldn't work or might confuse it?


  ld/ChangeLog:

	* ldlang.c (lang_process)[ENABLE_PLUGINS]: Move invocation of
	plugin_call_all_symbols_read to before setting of gc_sym_list, and
	open any new input files that may have been added during it.
	* ldmain.c (multiple_definition)[ENABLE_PLUGINS]: Call out to
	plugin_multiple_definition and let it have first say over what to do
	with the clashing definitions.
	* plugin.c (no_more_claiming): New boolean variable.
	(plugin_cached_allow_multiple_defs): Likewise.
	(plugin_call_claim_file): Don't do anything when no_more_claiming set.
	(plugin_call_all_symbols_read): Set it.  Disable link info
	"allow_multiple_definition" flag, but cache its value.
	(add_input_file): Implement.
	(add_input_library): Likewise.
	(set_extra_library_path): Likewise.
	(plugin_multiple_definition): New function.
	* plugin.h (plugin_multiple_definition): Add prototype.
	* testplug.c (addfile_enum_t): New enumerated typedef.
	(add_file_t): New struct typedef.
	(addfiles_list): New variable.
	(addfiles_tail_chain_ptr): Likewise.
	(record_add_file): New function.
	(parse_option): Parse "add:", "lib:" and "dir:" options and call it.
	(onall_symbols_read): Iterate the list of new files, libs and dirs,
	adding them.

  ld/testsuite/ChangeLog:

	* testsuite/ld-plugin/plugin-9.d: New testcase.
	* ld-plugin/plugin.exp: Invoke it.


    cheers,
      DaveK


>From 3e5b2e92af331115c82b9f3f943559be64ea1f4f Mon Sep 17 00:00:00 2001
From: Dave Korn <dave.korn.cygwin@gmail.com>
Date: Thu, 23 Sep 2010 03:23:19 +0100
Subject: [PATCH] Implement add input file, add input library and set extra library path
 callbacks.

  ld/ChangeLog:

	* ldlang.c (lang_process)[ENABLE_PLUGINS]: Move invocation of
	plugin_call_all_symbols_read to before setting of gc_sym_list, and
	open any new input files that may have been added during it.
	* ldmain.c (multiple_definition)[ENABLE_PLUGINS]: Call out to
	plugin_multiple_definition and let it have first say over what to do
	with the clashing definitions.
	* plugin.c (no_more_claiming): New boolean variable.
	(plugin_cached_allow_multiple_defs): Likewise.
	(plugin_call_claim_file): Don't do anything when no_more_claiming set.
	(plugin_call_all_symbols_read): Set it.  Disable link info
	"allow_multiple_definition" flag, but cache its value.
	(add_input_file): Implement.
	(add_input_library): Likewise.
	(set_extra_library_path): Likewise.
	(plugin_multiple_definition): New function.
	* plugin.h (plugin_multiple_definition): Add prototype.
	* testplug.c (addfile_enum_t): New enumerated typedef.
	(add_file_t): New struct typedef.
	(addfiles_list): New variable.
	(addfiles_tail_chain_ptr): Likewise.
	(record_add_file): New function.
	(parse_option): Parse "add:", "lib:" and "dir:" options and call it.
	(onall_symbols_read): Iterate the list of new files, libs and dirs,
	adding them.

  ld/testsuite/ChangeLog:

	* testsuite/ld-plugin/plugin-9.d: New testcase.
	* ld-plugin/plugin.exp: Invoke it.
---
 ld/ldlang.c                       |   18 +++++++--
 ld/ldmain.c                       |   13 ++++++-
 ld/plugin.c                       |   73 ++++++++++++++++++++++++++++++++++---
 ld/plugin.h                       |    6 +++
 ld/testplug.c                     |   59 ++++++++++++++++++++++++++++++
 ld/testsuite/ld-plugin/plugin-9.d |   35 ++++++++++++++++++
 ld/testsuite/ld-plugin/plugin.exp |    7 ++++
 7 files changed, 200 insertions(+), 11 deletions(-)

diff --git a/ld/ldlang.c b/ld/ldlang.c
index cb1417c..1310852 100644
--- a/ld/ldlang.c
+++ b/ld/ldlang.c
@@ -6317,6 +6317,9 @@ lang_relax_sections (bfd_boolean need_layout)
 void
 lang_process (void)
 {
+#ifdef ENABLE_PLUGINS
+  union lang_statement_union **listend;
+#endif /* ENABLE_PLUGINS */
   /* Finalize dynamic list.  */
   if (link_info.dynamic_list)
     lang_finalize_version_expr_head (&link_info.dynamic_list->head);
@@ -6339,19 +6342,26 @@ lang_process (void)
   current_target = default_target;
   open_input_bfds (statement_list.head, FALSE);
 
-  link_info.gc_sym_list = &entry_symbol;
-  if (entry_symbol.name == NULL)
-    link_info.gc_sym_list = ldlang_undef_chain_list_head;
-
 #ifdef ENABLE_PLUGINS
   /* Now all files are read, let the plugin(s) decide if there
      are any more to be added to the link before we call the
      emulation's after_open hook.  */
+  listend = statement_list.tail;
+  ASSERT (!*listend);
   if (plugin_call_all_symbols_read ())
     einfo (_("%P%F: %s: plugin reported error after all symbols read\n"),
       plugin_error_plugin ());
+  /* If any new files were added, they will be on the end of the
+     statement list, and we can open them now by getting open_input_bfds
+     to carry on from where it ended last time.  */
+  if (*listend)
+    open_input_bfds (*listend, FALSE);
 #endif /* ENABLE_PLUGINS */
 
+  link_info.gc_sym_list = &entry_symbol;
+  if (entry_symbol.name == NULL)
+    link_info.gc_sym_list = ldlang_undef_chain_list_head;
+
   ldemul_after_open ();
 
   bfd_section_already_linked_table_free ();
diff --git a/ld/ldmain.c b/ld/ldmain.c
index ed1738e..f1d6697 100644
--- a/ld/ldmain.c
+++ b/ld/ldmain.c
@@ -889,7 +889,7 @@ add_archive_element (struct bfd_link_info *info,
    multiple times.  */
 
 static bfd_boolean
-multiple_definition (struct bfd_link_info *info ATTRIBUTE_UNUSED,
+multiple_definition (struct bfd_link_info *info,
 		     const char *name,
 		     bfd *obfd,
 		     asection *osec,
@@ -898,6 +898,17 @@ multiple_definition (struct bfd_link_info *info ATTRIBUTE_UNUSED,
 		     asection *nsec,
 		     bfd_vma nval)
 {
+#ifdef ENABLE_PLUGINS
+  /* We may get called back even when --allow-multiple-definition is in
+     effect, as the plugin infrastructure needs to use this hook in
+     order to swap out IR-only symbols for real ones.  In that case,
+     it will let us know not to continue by returning TRUE even if this
+     is not an IR-only vs. non-IR symbol conflict.  */
+  if (plugin_multiple_definition (info, name, obfd, osec, oval, nbfd,
+	nsec, nval))
+    return TRUE;
+#endif /* ENABLE_PLUGINS */
+
   /* If either section has the output_section field set to
      bfd_abs_section_ptr, it means that the section is being
      discarded, and this is not really a multiple definition at all.
diff --git a/ld/plugin.c b/ld/plugin.c
index 4e8505d..a33e907 100644
--- a/ld/plugin.c
+++ b/ld/plugin.c
@@ -26,6 +26,9 @@
 #include "ld.h"
 #include "ldmain.h"
 #include "ldmisc.h"
+#include "ldexp.h"
+#include "ldlang.h"
+#include "ldfile.h"
 #include "plugin.h"
 #include "plugin-api.h"
 /* Don't include std headers until after config.h, sysdeps.h etc.,
@@ -119,6 +122,16 @@ static const char *error_plugin = NULL;
    cases when establishing symbol resolutions.  */
 static struct bfd_hash_table *non_ironly_hash = NULL;
 
+/* Set at all symbols read time, to avoid recursively offering the plugin
+   its own newly-added input files and libs to claim.  */
+static bfd_boolean no_more_claiming = FALSE;
+
+/* If the --allow-multiple-definition command-line option is active, we
+   have to disable it so that BFD always calls our hook, and simulate the
+   effect (when not resolving IR vs. real symbols) ourselves by ensuring
+   TRUE is returned from the hook.  */
+static bfd_boolean plugin_cached_allow_multiple_defs = FALSE;
+
 /* Helper to size leading part of tv array and set it up. */
 static size_t
 set_tv_header (struct ld_plugin_tv *tv)
@@ -347,6 +360,8 @@ plugin_call_claim_file (const struct ld_plugin_input_file *file, int *claimed)
 {
   plugin_t *curplug = plugins_list;
   *claimed = FALSE;
+  if (no_more_claiming)
+    return 0;
   while (curplug && !*claimed)
     {
       if (curplug->claim_file_handler)
@@ -367,6 +382,17 @@ int
 plugin_call_all_symbols_read (void)
 {
   plugin_t *curplug = plugins_list;
+
+  /* Disable any further file-claiming.  */
+  no_more_claiming = TRUE;
+
+  /* If --allow-multiple-definition is in effect, we need to disable it,
+     as the plugin infrastructure relies on the multiple_definition
+     callback to swap out the dummy IR-only BFDs for new real ones
+     when it starts opening the files added during this callback.  */
+  plugin_cached_allow_multiple_defs = link_info.allow_multiple_definition;
+  link_info.allow_multiple_definition = FALSE;
+
   while (curplug)
     {
       if (curplug->all_symbols_read_handler)
@@ -617,8 +643,9 @@ static enum ld_plugin_status
 add_input_file (const char *pathname)
 {
   ASSERT (called_plugin);
-  pathname = pathname;
-  return LDPS_ERR;
+  if (!lang_add_input_file (pathname, lang_input_file_is_file_enum, NULL))
+    return LDPS_ERR;
+  return LDPS_OK;
 }
 
 /* Add a new (real) library required by a plugin.  */
@@ -626,8 +653,9 @@ static enum ld_plugin_status
 add_input_library (const char *pathname)
 {
   ASSERT (called_plugin);
-  pathname = pathname;
-  return LDPS_ERR;
+  if (!lang_add_input_file (pathname, lang_input_file_is_l_enum, NULL))
+    return LDPS_ERR;
+  return LDPS_OK;
 }
 
 /* Set the extra library path to be used by libraries added via
@@ -636,8 +664,8 @@ static enum ld_plugin_status
 set_extra_library_path (const char *path)
 {
   ASSERT (called_plugin);
-  path = path;
-  return LDPS_ERR;
+  ldfile_add_library_path (path, FALSE);
+  return LDPS_OK;
 }
 
 /* Issue a diagnostic message from a plugin.  */
@@ -726,3 +754,36 @@ plugin_notice (struct bfd_link_info *info, const char *name, bfd *abfd,
   /* Continue with cref/nocrossref/trace-sym processing.  */
   return TRUE;
 }
+
+/* When we add new object files to the link at all symbols read time,
+   these contain the real code and symbols generated from the IR files,
+   and so duplicate all the definitions already supplied by the dummy
+   IR-only BFDs that we created at claim files time.  We use the linker's
+   multiple-definitions callback hook to fix up the clash, discarding
+   the symbol from the IR-only BFD in favour of the symbol from the
+   real BFD.  We return true if this was not-really-a-clash because
+   we've fixed it up, or anyway if --allow-multiple-definition was in
+   effect (before we disabled it to ensure we got called back).  */
+bfd_boolean
+plugin_multiple_definition (struct bfd_link_info *info, const char *name,
+		bfd *obfd, asection *osec, bfd_vma oval,
+		bfd *nbfd, asection *nsec, bfd_vma nval)
+{
+  osec = osec;
+  oval = oval;
+  if (is_ir_dummy_bfd (obfd))
+    {
+      struct bfd_link_hash_entry *blhe = bfd_link_hash_lookup (info->hash,
+					name, FALSE, FALSE, FALSE);
+      if (!blhe)
+	einfo (_("%P%X: %s: can't find IR symbol '%s'"), nbfd->filename,
+		name);
+      else if (blhe->type != bfd_link_hash_defined)
+	einfo (_("%P%x: %s: bad IR symbol type %d"), name, blhe->type);
+      /* Replace it with new details.  */
+      blhe->u.def.section = nsec;
+      blhe->u.def.value = nval;
+      return TRUE;
+    }
+  return plugin_cached_allow_multiple_defs;
+}
diff --git a/ld/plugin.h b/ld/plugin.h
index 5bbe93d..c21525b 100644
--- a/ld/plugin.h
+++ b/ld/plugin.h
@@ -64,5 +64,11 @@ extern bfd_boolean plugin_notice (struct bfd_link_info *info,
 		const char *name, bfd *abfd, asection *section,
 		bfd_vma value);
 
+/* Multiple-definition bfd linker callback hook.  */
+extern bfd_boolean plugin_multiple_definition (struct bfd_link_info *info,
+		const char *name,
+		bfd *obfd, asection *osec, bfd_vma oval,
+		bfd *nbfd, asection *nsec, bfd_vma nval);
+
 #endif /* !def GLD_PLUGIN_H */
 
diff --git a/ld/testplug.c b/ld/testplug.c
index ad1bdf6..1b775ed 100644
--- a/ld/testplug.c
+++ b/ld/testplug.c
@@ -48,6 +48,22 @@ typedef struct claim_file
   int n_syms_used;
 } claim_file_t;
 
+/* Types of things that can be added at all symbols read time.  */
+typedef enum addfile_enum
+{
+  ADD_FILE,
+  ADD_LIB,
+  ADD_DIR
+} addfile_enum_t;
+
+/* Struct for recording files to add to final link.  */
+typedef struct add_file
+{
+  struct add_file *next;
+  const char *name;
+  addfile_enum_t type;
+} add_file_t;
+
 /* Helper macro for defining array of transfer vector tags and names.  */
 #define ADDENTRY(tag) { tag, #tag }
 
@@ -117,6 +133,12 @@ static claim_file_t **claimfiles_tail_chain_ptr = &claimfiles_list;
 /* The last claimed file added to the list, for receiving syms.  */
 static claim_file_t *last_claimfile = NULL;
 
+/* The master list of all files to add to the final link.  */
+static add_file_t *addfiles_list = NULL;
+
+/* We keep a tail pointer for easy linking on the end.  */
+static add_file_t **addfiles_tail_chain_ptr = &addfiles_list;
+
 /* Add a new claimfile on the end of the chain.  */
 static enum ld_plugin_status
 record_claim_file (const char *file)
@@ -135,6 +157,22 @@ record_claim_file (const char *file)
   return LDPS_OK;
 }
 
+/* Add a new addfile on the end of the chain.  */
+static enum ld_plugin_status
+record_add_file (const char *file, addfile_enum_t type)
+{
+  add_file_t *newfile;
+
+  newfile = xmalloc (sizeof *newfile);
+  newfile->next = NULL;
+  newfile->name = file;
+  newfile->type = type;;
+  /* Chain it on the end of the list.  */
+  *addfiles_tail_chain_ptr = newfile;
+  addfiles_tail_chain_ptr = &newfile->next;
+  return LDPS_OK;
+}
+
 /* Parse a command-line argument string into a symbol definition.
    Symbol-strings follow the colon-separated format:
 	NAME:VERSION:def:vis:size:COMDATKEY
@@ -277,6 +315,12 @@ parse_option (const char *opt)
     return record_claim_file (opt + 6);
   else if (!strncmp ("sym:", opt, 4))
     return record_claimed_file_symbol (opt + 4);
+  else if (!strncmp ("add:", opt, 4))
+    return record_add_file (opt + 4, ADD_FILE);
+  else if (!strncmp ("lib:", opt, 4))
+    return record_add_file (opt + 4, ADD_LIB);
+  else if (!strncmp ("dir:", opt, 4))
+    return record_add_file (opt + 4, ADD_DIR);
   else if (!strcmp ("dumpresolutions", opt))
     dumpresolutions = TRUE;
   else
@@ -514,6 +558,7 @@ onall_symbols_read (void)
       "LDPR_RESOLVED_DYN",
     };
   claim_file_t *claimfile = dumpresolutions ? claimfiles_list : NULL;
+  add_file_t *addfile = addfiles_list;
   TV_MESSAGE (LDPL_INFO, "hook called: all symbols read.\n");
   for ( ; claimfile; claimfile = claimfile->next)
     {
@@ -535,6 +580,20 @@ onall_symbols_read (void)
 					      : "",
 		resolutions[claimfile->symbols[n].resolution]);
     }
+  for ( ; addfile ; addfile = addfile->next)
+    {
+      enum ld_plugin_status rv;
+      if (addfile->type == ADD_LIB && tv_add_input_library)
+	rv = (*tv_add_input_library) (addfile->name);
+      else if (addfile->type == ADD_FILE && tv_add_input_file)
+	rv = (*tv_add_input_file) (addfile->name);
+      else if (addfile->type == ADD_DIR && tv_set_extra_library_path)
+	rv = (*tv_set_extra_library_path) (addfile->name);
+      else
+	rv = LDPS_ERR;
+      if (rv != LDPS_OK)
+	return rv;
+    }
   fflush (NULL);
   return all_symbols_read_ret;
 }
diff --git a/ld/testsuite/ld-plugin/plugin-9.d b/ld/testsuite/ld-plugin/plugin-9.d
new file mode 100644
index 0000000..b74f4a6
--- /dev/null
+++ b/ld/testsuite/ld-plugin/plugin-9.d
@@ -0,0 +1,35 @@
+Hello from testplugin.
+tv\[0\]: LDPT_MESSAGE func@0x.*
+tv\[1\]: LDPT_API_VERSION value        0x1 \(1\)
+tv\[2\]: LDPT_GNU_LD_VERSION value       0x.*
+tv\[3\]: LDPT_LINKER_OUTPUT value        0x1 \(1\)
+tv\[4\]: LDPT_OUTPUT_NAME 'tmpdir/main.x'
+tv\[5\]: LDPT_REGISTER_CLAIM_FILE_HOOK func@0x.*
+tv\[6\]: LDPT_REGISTER_ALL_SYMBOLS_READ_HOOK func@0x.*
+tv\[7\]: LDPT_REGISTER_CLEANUP_HOOK func@0x.*
+tv\[8\]: LDPT_ADD_SYMBOLS func@0x.*
+tv\[9\]: LDPT_GET_INPUT_FILE func@0x.*
+tv\[10\]: LDPT_RELEASE_INPUT_FILE func@0x.*
+tv\[11\]: LDPT_GET_SYMBOLS func@0x.*
+tv\[12\]: LDPT_ADD_INPUT_FILE func@0x.*
+tv\[13\]: LDPT_ADD_INPUT_LIBRARY func@0x.*
+tv\[14\]: LDPT_SET_EXTRA_LIBRARY_PATH func@0x.*
+tv\[15\]: LDPT_OPTION 'registerclaimfile'
+tv\[16\]: LDPT_OPTION 'registerallsymbolsread'
+tv\[17\]: LDPT_OPTION 'registercleanup'
+tv\[18\]: LDPT_OPTION 'claim:tmpdir/func.o'
+tv\[19\]: LDPT_OPTION 'sym:_?func::0:0:0'
+tv\[20\]: LDPT_OPTION 'sym:_?func2::0:0:0'
+tv\[21\]: LDPT_OPTION 'dumpresolutions'
+tv\[22\]: LDPT_OPTION 'add:tmpdir/func.o'
+tv\[23\]: LDPT_NULL value        0x0 \(0\)
+#...
+hook called: claim_file tmpdir/main.o \[@0/.* not claimed
+hook called: claim_file tmpdir/func.o \[@0/.* CLAIMED
+hook called: claim_file tmpdir/text.o \[@0/.* not claimed
+#...
+hook called: all symbols read.
+Sym: '_?func' Resolution: LDPR_PREVAILING_DEF
+Sym: '_?func2' Resolution: LDPR_PREVAILING_DEF_IRONLY
+hook called: cleanup.
+#...
diff --git a/ld/testsuite/ld-plugin/plugin.exp b/ld/testsuite/ld-plugin/plugin.exp
index 6ed2787..32f396c 100644
--- a/ld/testsuite/ld-plugin/plugin.exp
+++ b/ld/testsuite/ld-plugin/plugin.exp
@@ -99,6 +99,13 @@ set plugin_tests [list \
 			-plugin-arg sym:${_}func2::0:0:0 \
 			-plugin-arg dumpresolutions \
     $testobjfiles $libs" "" "" {{ld plugin-8.d}} "main.x" ] \
+    [list "plugin claimfile replace file" "-plugin $plugin_path $regclm \
+			$regas $regcln -plugin-arg claim:tmpdir/func.o \
+			-plugin-arg sym:${_}func::0:0:0 \
+			-plugin-arg sym:${_}func2::0:0:0 \
+			-plugin-arg dumpresolutions \
+			-plugin-arg add:tmpdir/func.o \
+    $testobjfiles $libs" "" "" {{ld plugin-9.d}} "main.x" ] \
 ]
 
 if { $failed_compile != 0 } {