windows ci failed with STATUS_DLL_INIT_FAILED

[evenyag]

What type of bug is this?

Unexpected error

What subsystems are affected?

Standalone mode

Minimal reproduce step

Run the nightly CI
https://github.com./GreptimeTeam/greptimedb/actions/runs/9009678033/job/24754350747

What did you expect to see?

The CI is passed

What did you see instead?

The CI is failed

What operating system did you use?

Windows Server 2022

What version of GreptimeDB did you use?

latest

Relevant log output and stack trace

Caused by:
  for `benchmarks::bin/nyc-taxi`, command `'C:\a\greptimedb\greptimedb\target\debug\deps\nyc_taxi-1c449a394a7932c1.exe' --list --format terse` exited with code 0xc0000142: A dynamic link library (DLL) initialization routine failed. (os error 1114)


     Running `target\debug\sqlness-runner.exe`
error: process didn't exit successfully: `target\debug\sqlness-runner.exe` (exit code: 0xc0000142, STATUS_DLL_INIT_FAILED)
Error: Process completed with exit code 1.

[evenyag]

On Windows Server 2022 there are some strange DDL issues.

1434:0d10 @ 09019093 - LdrpInitializeNode - ERROR: Init routine 00007FFDA6925700 for DLL "C:\Windows\SYSTEM32\vertdll.dll" failed during DLL_PROCESS_ATTACH
1434:0d10 @ 09019093 - LdrpInitializeProcess - ERROR: Running the init routines of the executable's static imports failed with status 0xc0000142
1434:0d10 @ 09019093 - _LdrpInitialize - ERROR: Process initialization failed with status 0xc0000142

After adding vertdll.dll from Windows 10 to target/debug/, the sqlness.exe could be executed. So it seems that the binary has some issues using the vertdll.dll on Windows Server 2022.

[evenyag]

The dumpbin command shows that we uses WaitOnAddress from the vertdll.dll

 .\dumpbin.exe -imports sqlness-runner.exe

Outputs:

vertdll.dll
            141C09710 Import Address Table
            1423918D8 Import Name Table
                    0 time date stamp
                    0 Index of first forwarder reference

                        7 WaitOnAddress
                        9 WakeByAddressSingle
                        8 WakeByAddressAll

I wonder if it is related to this issue rust-lang/rust#123999 as rust-lang/rust#124019 mentioned WaitOnAddress

[evenyag]

Some tests are failed on Windows Server 2019. We got a permission denied error while deleting a file. But the file is actually deleted.

2024-05-11T12:00:57.565505Z  INFO common_telemetry::logging: logs dir = /tmp/__unittest_logs
2024-05-11T12:00:57.566327Z  INFO mito2::cache::file_cache::tests: temp dir is C:\Users\ADMINI~1\AppData\Local\Temp\2\T8ZIFB
2024-05-11T12:00:57.777286Z DEBUG mito2::cache::file_cache: deleted file files/8589934592000.67b7aad5-1616-491e-9301-55e6dadfd2e3.parquet
2024-05-11T12:00:57.778244Z  WARN mito2::cache::file_cache: Failed to delete a cached file files/8589934592000.67b7aad5-1616-491e-9301-55e6dadfd2e3.parquet err=PermissionDenied (permanent) at delete => permission denied

Context:
   service: fs
   path: files/8589934592000.67b7aad5-1616-491e-9301-55e6dadfd2e3.parquet

Source:
   Access is denied. (os error 5)

[evenyag]

Let's wait for the next release to see if everything works fine.

[evenyag]

Maybe another related issue: nextest-rs/nextest#1493

[evenyag]

The error still exists in the release action......
https://github.com./GreptimeTeam/greptimedb/actions/runs/9114277736/job/25057956421#logs

[evenyag]

The error still exists in the release action...... https://github.com./GreptimeTeam/greptimedb/actions/runs/9114277736/job/25057956421#logs

Added a workaround for this in #3969

[evenyag]

Let's keep this open until nextest-rs/nextest#1493 is fixed so we can upgrade the nextest that contains that patch.