Passing -lto-embed-bitcode=optimized to the lld gives a corrupted .llvmbc section.
#84395
Comments
|
By the way, I'm happy to fix this and raise a PR if we can figure out how to fix it :) |
|
Looked into this a little more this morning. Here if the bytes are indeed LLVM bitcode boundaries, then we have 4 modules in the I used Then used So this confirms that rustc is encoding many modules of bitcode into the binary. My guess is that Here are the link args rustc used: Notice this part: The three amigos... It's still not clear to me why their bitcode is retained. |
|
I got it locally setup using: I tried passing Looking at https://reviews.llvm.org/D68213 which implemented this if I read it correctly it seems to be the case that |
|
@bjorn3 In the above do you get a "multi-module" bitcode section too?
I've been pouring over this code recently too (but it's changed a little since the review you linked). The linker plugin supports three embedding modes: For |
Yes
In FunctionImporter Importer(CombinedIndex, ModuleLoader);
if (Error Err = Importer.importFunctions(Mod, ImportList).takeError())
return Err;Or at least there was as of https://reviews.llvm.org/D68213. |
|
It's moved around a little, but it looks like the ordering is:
I also just checked what building C code with different compilation units does. There is still a single module in the output binary's bitcode: So we don't see 3 modules here, only the one for the post-LTO bitcode. Good. So why is Rust keeping the intermediate bitcodes? I wonder if it is adding section flags that cause them to be retained or something like that? |
Just to add, this may not be correct, since |
|
Can you try to put a function in one module and then call it from another module. This function should be inlined by LTO. If on of the the |
|
world.map.4.ll may be the allocator shim. Does it contain functions of form |
|
Looks like you may be right: |
|
I looked at the llvm ir of all the extracted bitcode files and it seems to be completely unoptimized. |
I think you are also right about this. It looks like we are not seeing any post-LTO bitcode. The definition of the "other" sting and it's pointer is here in And its use is in a function not So the linker plugin isn't doing anything?! Surely not? |
|
I think the difference between clang and rustc is that rustc directly puts llvm bitcode into the "object" files while clang believe wraps them in elf files. |
|
I don't think that's true. It's one of the things I investigated earlier: |
|
Well, I'm baffled. I put prints all over libLTO (and an abort, as rustc consumes stderr otherwise): And we get: Which tells us that It occurs to me that the std rlibs have to be "hybrid" files: executable code, but with a It might have made sense if the extra modules we were seeing were just the So all in all, I'm confused. |
|
And looking at the binary, |
camelid
added
the
A-linkage
|
Hi! Any update about this issue? |
|
Hi, Encountered the same issue and looked into it, found out a possible change to rustc that will allow "-lto-embed-bitcode=optimized" to work. First of all the reason it doesn't work is that rustc generate bitcode modules for thinLTO, which in turn ld.lld recognize and passes via the thinLTO backend. Basically, "lto-embed-bitcode" doesn't work when working with thinLTO, which is the default and only way rustc works with modules when passing them to Clinker-plugin-lto. Where ThinBuffer is actually, running the following llvm pass Changing (only for call site in write.rs:codegen) to go through "createBitcodeWriterPass(OS)" pass instead generated bitcode modules without thinLTO info. I suggest to add a config option to rustc to select which pass to run. Which would need to be used for "lto-embed-bitcode=optimized" |
…, r=davidtwco Allow to disable thinLTO buffer to support lto-embed-bitcode lld feature Hello This change is to fix issue (rust-lang#84395) in which passing "-lto-embed-bitcode=optimized" to lld when linking rust code via linker-plugin-lto doesn't produce the expected result. Instead of emitting a single unified module into a llvmbc section of the linked elf, it emits multiple submodules. This is caused because rustc emits the BC modules after running llvm `createWriteThinLTOBitcodePass` pass. Which in turn triggers a thinLTO linkage and causes the said issue. This patch allows via compiler flag (-Cemit-thin-lto=<bool>) to select between running `createWriteThinLTOBitcodePass` and `createBitcodeWriterPass`. Note this pattern of selecting between those 2 passes is common inside of LLVM code. The default is to match the old behavior.
Hi everyone,
(This issue is based on this SO post)
libLTO has the option to embed the post-merged-and-optimised (i.e. final) bitcode into the end binary. This is done with the
-lto-embed-bitcode=optimizedoption tolld.Example of the use of this option with clang:
Equivalent in the Rust world:
If I search the dumped section for the bitcode magic header bytes,
0x4243c0de, there are lots of hits. If I add-C codegen-units=1toRUSTFLAGSthen there are then fewer hits (exactly two).It looks to me (although I can't be sure) like rustc is invoking the linker in such a way that the
.llvmbcsections of the intermediate objects are not being discarded post-LTO, but instead being concatenated back to back (as linkers do) with the.llvmbcsection of the post-LTO bitcode. So.llvmbcwould contain many modules, but it should contain only the post-LTO bitcode module.Assuming this is the case, this is problematic: it's not trivial to split the concatenated bitstreams apart (the magic bytes cannot be used as a reliable delimiter, as those bytes may appear in other unrelated contexts), and even if we could, we wouldn't know which of the resulting modules was the post-LTO one.
The text was updated successfully, but these errors were encountered: