We recently upgraded our application using U2.Data.Client from version 3.1.1.0 to 3.1.2.0, and we are encountering a different kind of issue with multithreading.
Issue on Version 3.1.1.0
Previously, in version 3.1.1.0, our server would spike the CPU usage to 100% and remain at that level. Upon inspecting the thread stack traces, we observed that all threads were stuck on modifying an internal Dictionary inside the ThreadPoolManager:
000000cf40c7de30 00007ffc2322e241 System.Collections.Generic.Dictionary`2[[System.Int32, mscorlib],[System.__Canon, mscorlib]].Insert(Int32, System.__Canon, Boolean)
000000cf40c7dec0 00007ffbce88c056 U2.Data.Client.U2PoolManager+h.c(System.Object)
000000cf40c7df10 00007ffbce8725bb U2.Data.Client.U2PoolManager.a(U2.Data.Client.U2Connection, System.String ByRef, d ByRef, System.Object ByRef)
Issue After Upgrading to Version 3.1.2.0
After upgrading to 3.1.2.0, the CPU no longer spikes to 100%, but all requests are now stuck waiting on:
0000000afd67e248 00007ff9dbed6be4 [HelperMethodFrame_1OBJ: 0000000afd67e248] System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
0000000afd67e370 00007ff9c93acf1c System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
0000000afd67e3a0 00007ff9c93aceef System.Threading.WaitHandle.WaitOne(Int32, Boolean)
0000000afd67e3e0 00007ff989bac7b3 U2.Data.Client.U2PoolManager.Open(U2.Data.Client.U2Connection, System.String ByRef, U2ConnSettings ByRef, System.Object ByRef)
0000000afd67e580 00007ff989bac392 U2.Data.Client.U2Connection.Open()
Additionally, we are now encountering an exception:
System.ArgumentException: An item with the same key has already been added.
at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource)
at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
at U2.Data.Client.U2PoolManager.Open(U2Connection connection, String& szConnectionString, U2ConnSettings& ppSettings, Object& ppConn)
at U2.Data.Client.U2Connection.Open()
Observations
-
The original issue in 3.1.1.0 seems related to a non-thread-safe Dictionary, causing high CPU usage as multiple threads attempted to modify it simultaneously.
-
In 3.1.2.0, instead of high CPU usage, we now see all requests waiting on WaitHandle.WaitOne, potentially indicating a lock contention.
-
The System.ArgumentException suggests that the dictionary is still being accessed in a non-thread-safe way, leading to duplicate key insertions.
Workaround Implemented
To mitigate the issue, we have implemented thread locking to prevent concurrent modifications to the internal dictionary. However, we are looking for a more efficient approach to handle this problem without causing excessive thread blocking or lock contention.
References
This article describes similar behavior in .NET applications related to dictionary thread-safety issues: Fix High CPU Caused by Dictionary Thread-Safe Issue
Has anyone else experienced similar issues after upgrading? Are there alternative solutions to ensure thread safety while maintaining performance? Any suggestions or workarounds would be greatly appreciated!