Rebase Doesn't Change Data Symbol Address

[utkonos]

Version and Platform (required):

• Binary Ninja Version: 5.0.7202-dev
• OS: macOS
• OS Version: 15.4
• CPU Architecture: x64

Bug Description:
You all fixed a crash that was happening right at rebase (Thanks!!). Crash is gone, but there is now a bug in rebase itself. The data symbol address for the payload DLL does not rebase with the rest of the database. All references to that data symbol are off by difference in old and new base.

Steps To Reproduce:

1. Open database: bright moon dreams happily (this is new version uploaded just now to portal)
2. Navigate to 0x27a00c9 and take note of how payload looks before rebase.
3. Double click on payload (beachball here on my UI and it lands at the end of the string which may be intentional but odd with large string.)
4. Navigate to 0x27a0dde (this is above the start of the string so you don't have to scroll for a million years and whatnot).
5. Analysis menu -> Rebase
6. Change 0x27a0000 to 0x2780000
7. Click Accept
8. Navigate to 0x27800c9
9. Note how 0x3c became 0x2003c

Expected Behavior:
The data symbol addresses rebase with everything else.

Screenshots/Video Recording:
Before:

After:

[utkonos]

Note: that DanaBot DLL contains a curse or something 🤣 or it is so heavy that binary ninja is having trouble picking it up and moving it. Maybe he adversary who compiled the DLL had too many fat electrons in their computer at the time and lots of them got trapped in this DLL.

[zznop]

Edit: disregard, I was using the wrong API. That just showed that you defined the variable, not the value of the variable

[utkonos]

@zznop I never set that. It's something that BN does automatically. Here is a screenshot of it setting ebx automatically to 0x6 (i just opened the shellcode in a new database and picked 0x0 as image base this time). I think there is some bug in the interaction between detecting the global ebx and when i make a user defined data symbol for the DLL payload.

[utkonos]

I don't know because I can't see it (or I don't know how to), but it feels like whatever analysis component it is that detects that global pointer regiater is entering it in the database as user defined? And then everything downstream from it is being considered user deifned and therefore not being rebased?

[zznop]

Ok, thanks I'll keep digging

[utkonos]

@zznop I think I can at least isolate the effects. I uploaded the bare shellcode again here: fierce dolphin jumps quietly

One sec and I will type out each step from open with options so you can see this:

Then after rebase:

This is a different location because I didn't do any annotations other than making the data symbol, but this should show you where the problem is located, maybe?

[utkonos]

Steps:

1. Open with options
2. Set entry point offset: 0x0
3. Set platform: windows-x86
4. Set initial analysis hold
5. Navigate to 0xdea
6. Right click change type to: char payload[0x581000]
7. Wait for beachballs.
8. Navigate to 0x84
9. Change function type: void load_payload(void* shellcode_base @ ebx)
10. Navigate to 0xe1
11. Notice strangeness here.
12. Rebase to 0x27a0000
13. Now look at 0x27a00e1

This is different from what was going on in my original database, however. BTW: does the history in my database show where/how that value 0x27a0006 was set?

[utkonos]

I am not able to reproduce the precise problem just using dev. it has a different problem shown by the above steps. I think it may have something to do with the database being created in stable. I am starting with a fresh database in stable to see if I can get a list of steps to lead you precisely to the effect shown in the very first screenshots.

[utkonos]

Here is the text of the History: fierce robot flies softly

I grepped it for 0x27a0006 and there is no mention of making a user defined variable value and the only mention of this address and ebx is in my comments (these are comments from before I realized that BN was detecting this address on its own as a global pointer register.

I feel like whatever is setting the global pointer register value is doing something unexpected:

[utkonos]

@zznop I found something:

This value is the same before and after rebase:

[zznop]

Yeah ebx is defined as a global global pointer register per our default x86 calling convention. For global pointer registers, Binja will attempt to track the value and persist it across functions.

In the case of this sample, ebx is not being used as global register. It's being used for whatever the author wants (not surprising since this is shellcode). For example, at 0x2780072 the author decided to use ebx for the call/pop trick to do PC-relative addressing.

call    $+5
pop     ebx
sub     ebx, 0x71

In this case, we don't want Binja to treat ebx as a global pointer register. The value for ebx might be right in some cases, but in other cases it will need to be overridden. With shellcode (or anything written in assembly) standard calling conventions go out the window. The state of ebx should not be tracked across functions.

[zznop]

As a workaround I suggest trying this (in the python console)

bv.set_user_global_pointer_value(Undetermined())

[utkonos]

@zznop yes, totally correct. In some bent way, the call/pop trick here:

Essentially sets ebx as the input parameter for find_kernel32 and almost all its callees except for when a callee pushes ebx to the stack to save it for pop on return. That ebx is then also the "input parameter" scare quotes intentional, for load_payload where the funky stuff is going on.

[utkonos]

@zznop Ok, that workaround did the trick. Also, I tried reopening with options and setting this to 2000000:

It still sets the global pointer automagically:

[utkonos]

When I look at that value after the database is open, it has been set back to 2 from 2000000

[utkonos]

I tried 0 and -1 and they don't work

[utkonos]

Also interesting, is right after open with options, a global pointer is found and set, but the value of bv.global_pointer_value
is <undetermined>

[utkonos]

Back to the problem database, not a new one:

A user global pointer is not set:

And that wierdness I pointed out way back at the top is happening after rebase:

Perhaps something in there is thinking it is a user global pointer and rebase is broken because of that, but the global pointer is really auto and there is a bug somewhere that's preventing it from being rebased correctly.

@zznop I think this is near the core issue ☝

[utkonos]

I got it. Here is the series of steps to see the bug:

1. Open the database.
2. Navigate to 0x27a00c9
3. Python console: bv.global_pointer_value
4. Observe <const ptr 0x27a0006>
5. Python console: bv.user_global_pointer_value_set
6. Observe False
7. Analysis menu -> Rebase
8. Change 0x27a0000 to 0x2780000
9. Click Accept
10. Observe: 027800c9 8b773c mov esi, dword [edi+0x3c] {payload[0x2003c]} problem with payload[0x2003c]
11. Check global pointer again.
12. Both False and same value as before.

@zznop This isolates/demonstrates the bug.

[zznop]

ebx should not be tracked as a global pointer for this binary at all. It is not a global pointer register by definition as this binary doesn't use standard x86 calling conventions and sets ebx all over the place.

bv.user_global_pointer_value_set just tells you whether or not the user overrode the global pointer value. It will also return false unless you as the user overrides its value. However, even if the user doesn't, Binja will attempt to track changes to the global pointer register inter-procedurally until it hits analysis.limits.maxGlobalPointerValueUpdates. Typically compilers will only set ebx once for normal x86 programs and the program will use it for global memory accesses. ebx is a callee-saved register and therefore the value of ebx is preserved. In your shellcode sample, ebx is clobbered in multiple places and is not used as a global pointer register whatsoever.

This is a case where the calling convention used for this function is incorrect for your sample. The concept of a global pointer register doesn't apply to this shellcode, and we just need to disable the global pointer value tracking. This can be done by setting analysis.limits.maxGlobalPointerValueUpdates to 0 if you're analyzing the binary for the first time. In your case, this is a bndb, so you need to just force it to be undetermined with bv.set_user_global_pointer_value(Undetermined()), because Binja saves the global pointer value to the bndb.

[utkonos]

@zznop Ok. I think I have found three separate bugs:

1. Setting analysis.limits.maxGlobalPointerValueUpdates to zero when opening with options is ignored but it should be followed.
2. Rebase does not change auto global pointer value (I understand it shouldn't be set in the first place, but rebase should change it if it's there)
3. In this shellcode ebx is clobbered in multiple functions, therefore GP should just be auto and undetermined.

Does this list sound correct to you?

[zznop]

After supplying bv.set_user_global_pointer_value(Undetermined()) on your bndb, this is what I get

Also I found that you can do this without interacting with the python console (via the command palette):

[utkonos]

Yes, and that looks good. And if I try bv.clear_user_global_pointer_value() after that, it goes back to crazy stuff.

[utkonos]

@zznop Here is a fourth bug to the list:

A global pointer value is set even when the user value is set to undetermined:

You can still see it in that function properties in the function I have named main

[utkonos]

Global pointer bugs:

• Setting analysis.limits.maxGlobalPointerValueUpdates to zero when opening with options is ignored but it should be followed.
• Rebase does not change auto global pointer value (I understand it shouldn't be set in the first place, but rebase should change it if it's there)
• In this shellcode ebx is clobbered in multiple functions, therefore GP should just be auto and undetermined.
• A global pointer value is set even when the user value is set to undetermined

[zznop]

@zznop Ok. I think I have found three separate bugs:

1. Setting analysis.limits.maxGlobalPointerValueUpdates to zero when opening with options is ignored but it should be followed.
2. Rebase does not change auto global pointer value (I understand it shouldn't be set in the first place, but rebase should change it if it's there)
3. In this shellcode ebx is clobbered in multiple functions, therefore GP should just be auto and undetermined.

Does this list sound correct to you?

1 - When I open a new binary with analysis.limits.maxGlobalPointerValueUpdates set to 0 bv.global_pointer_value returns undetermined. This setting only applies to initial analysis though. If you're reopening an existing bndb, then the global pointer value is pulled from the database, and needs overridden with bv.set_user_global_pointer_value(Undetermined()).

2 - This is possibly a problem, if the global pointer register is set relative to the program counter. I will look into this.

3 - I wouldn't call this a bug. From Binja's perspective there should only be 1 global pointer value. It uses a voting system to determine which value to use (hence the analysis.globalPointerValueMinimumMajorityVotes setting). Perhaps we could add a heuristic to just force it to undetermined if it's clobbered too much. But again, this generally doesn't occur in programs that truly use a global pointer register.

4 - "A global pointer value is set even when the user value is set to undetermined" - I'm not seeing this behavior? When we set the pointer value to undetermined, Binja is honoring it. Not sure what you mean by this.

[utkonos]

@zznop Here is a video of #1 you can see that I set analysis.limits.maxGlobalPointerValueUpdates during open with options, not open an existing database. At then end I show the function properties with the global pointer set to ebx with the detected value from analysis. I then show that the API returns a totally different value of undetermined.

One sec and I will make a video of #4

The first video is too large for Github upload, but here it is in the portal: brave wizard reads boldly

[utkonos]

Ok, I made a video showing #4 where I just open the already existing database. And the GP is always there. It even gets rebased properly when the API has set an undetermined user global pointer.

portal ulpload: fierce dragon flows boldly

[utkonos]

And, yes, I can see how #3 could be considered an enhancement rather than a bug.

[utkonos]

Also, for #1 , it definitely gets set back to 10 and the open with options ignores that user setting option. After opening, take a look at the settings, it shows 10 again.

[zznop]

I watched your videos and see what you mean. #1 and #4 are really the same issue. The "Function Properties" UI shows the global pointer value that was identified by Binja, but doesn't tell you that it has been overridden by the user or that the analysis.limits.maxGlobalPointerValueUpdates was set to 0 and therefore it's actually undetermined.

In both cases though, analysis respected your desired behavior and the global pointer register remained undetermined (as shown from the Python console output). We just need to update the Function Properties dialog to show more information so it is known that the function isn't using the Binja-identified gp value in that function.

[utkonos]

For #1 yes it does respect the setting. But take a look in the settings after it has opened. The setting is 10 not 0 as it should be if it was opened with that option (it respected it, but changed it back to the default as well).

[utkonos]

@zznop thanks for sticking with me btw! So here is a revised list:

Bugs:

1. Open with options analysis.limits.maxGlobalPointerValueUpdates=0 is set back to 10 after the file is open.
2. Function Properties needs to be updated to show correct information.
3. Rebase does not rebase an auto GP when it should.

And one enhancement:

Better handling of GP if it is clobbered past a certain threshold to set it to undetermined automatically.

[zznop]

After you've opened the binary, if you open settings from the command palette the values will not reflect the values of the opened binary view. The settings dialog will show global settings, so you have the option to make analysis.limits.maxGlobalPointerValueUpdates 0 for all binaries (if you desired). It's showing 10 because you only overrode that setting for a single binary through Open with Options.

Also, I tested #3 with MIPS binaries that actually use a global pointer register. After rebase, the global pointer value is being updated correctly. We likely don't see that occur on your shellcode binary because it's reaching analysis.limits.maxGlobalPointerValueUpdates

I agree with #2 (not ruling out #3 though, just not seeing that behavior on normal binaries that use a GP)

[utkonos]

Thanks for all your help! And with the workaround my database looks the same as x64dbg now:

And screenshots of the database look nice again:

[psifertex]

Is this issue resolved?

[zznop]

I think the primary issue has been resolved. But I created new issue for the function properties dialog: #6675

[utkonos]

I'm using 5.0.7265-dev and I still observe the one part of this issue still happening: when I do bv.clear_user_global_pointer_value() I can see <const ptr 0x27a0006> as the response to bv.global_pointer_value. This is not a value that I set. When I rebase, that value remains the same as it was before rebase.

And after rebase, the strange stuff happens in the database view:

Then if I rebase that value myself: bv.set_user_global_pointer_value(ConstantPointerRegisterValue(0x2780006)) the database looks fine again:

However, when I use undetermined, it looks better: bv.set_user_global_pointer_value(Undetermined())

I think there is a bug with rebase that is not rebasing that automatically generated global pointer value.

[utkonos]

@zznop @psifertex I have a workaround for the original issue that makes the database look ideal. However, it looks like there still is a bug.

[zznop]

It's not that it's "not rebasing" the global pointer value. It's that the global pointer value isn't computed by PC-relative code in this shellcode (because this shellcode doesn't use a global pointer at all and is using ebx as a GPR). When you rebase the binary, the global pointer analysis runs again. In the case of this binary, the base address of the binary doesn't effect the "global pointer value".

I have tested multiple MIPS binaries that actually use a GP and where the value of the global pointer is computed via PC-relative code and found that when rebasing, the global pointer is updated correctly.

[utkonos]

I must be explaining this in a poor manner. Let me try again.

If I open a file with options, and one of the options is to put the value 0x27a0000 in this location to set the image base:

Then later down the line, I need to rebase because I'm using x64dbg and I've restarted my analysis over there in the VM.

If after I have rebased to 0x2780000 and I look at bv.global_pointer_value and the response is <const ptr 0x27a0006> I am confused because this value did not change when rebase occurred.

I totally understand that there are under the hood reasons for this, but it's still an astonishing outcome from the user's perspective according to the principle of least astonishment.

I respect that this is a won't fix, I just want to make sure that at least my perspective is understood.

[xusheng6]

A bit late to the party and I wish to clarify a few things:

1. If you call set_user_global_pointer_value to set a user global pointer value, it uses that value to override the global pointer value from auto-analysis.
2. If you call clear_user_global_pointer_value, it removes the user global pointer value, and then enables the auto-analysis to figure out what is the best global pointer value. The auto analysis can decide that the global pointer value should be anything it sees fit, or undertermined. So, if you really wish to set the value to undetermined, you should follow what Brandon is doing and do bv.set_user_global_pointer_value(Undetermined()). clear_user_global_pointer_value will not give you the same result
3. The current global pointer value can also be observed from the triage view

4. The analysis.globalPointerValueMinimumMajorityVotes is a number, and it is the minimum number of votes for a majority vote to be accepted. Let us say we have 10 functions 5 functions cast a vote on the same value, then this value should be used as the majority value. However, what if only one functions casts a vote? In that case, it may or may not be what we want. So the default value of analysis.globalPointerValueMinimumMajorityVotes is set to 2. Which means that when there are at least two functions that cast a vote on the same value, the value can be trusted.

I have not read the full thread yet, and will do so when possible. Hopefully what I wrote above can be helpful