NT Crash Thread

[John Adams]

Crash - though it's the heap thing, so according to Lokked's previous posts, it may not be in the functions listed (which seems odd to be, but I'll go with it).

Code: Select all

 	ntdll.dll!774ce725()	Unknown
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	ntdll.dll!774cf659()	Unknown
 	ntdll.dll!77433cfe()	Unknown
 	ntdll.dll!77433cfe()	Unknown
>	WorldServer.exe!_heap_alloc_base(unsigned int size) Line 57	C
 	WorldServer.exe!_heap_alloc_dbg_impl(unsigned int nSize, int nBlockUse, const char * szFileName, int nLine, int * errno_tmp) Line 431	C++
 	WorldServer.exe!_nh_malloc_dbg_impl(unsigned int nSize, int nhFlag, int nBlockUse, const char * szFileName, int nLine, int * errno_tmp) Line 239	C++
 	WorldServer.exe!_nh_malloc_dbg(unsigned int nSize, int nhFlag, int nBlockUse, const char * szFileName, int nLine) Line 302	C++
 	WorldServer.exe!malloc(unsigned int nSize) Line 56	C++
 	WorldServer.exe!SOEProtocolData::SetData(PacketStruct * packet_struct) Line 100	C++
 	WorldServer.exe!PacketStruct::Serialize() Line 724	C++
 	WorldServer.exe!ChunkServer::HandleClientTrainingBegin(const std::shared_ptr<Client> & client) Line 5224	C++
 	WorldServer.exe!ChunkServer::ProcessPackets() Line 403	C++
 	WorldServer.exe!ChunkPacketThread(void * data) Line 134	C++
 	WorldServer.exe!ThreadRun(void * arg) Line 77	C++
 	WorldServer.exe!_callthreadstart() Line 255	C
 	WorldServer.exe!_threadstart(void * ptd) Line 239	C
 	kernel32.dll!7519338a()	Unknown
 	ntdll.dll!77439f72()	Unknown
 	ntdll.dll!77439f45()	Unknown

I've seen this line being called out more than once; so I think it deserves immediate attention, just in case it's not so "random" and heap'ish

Code: Select all

client->QueuePacket(out_nontrainable_abilities->Serialize());

Console, if it helps:
[quote]13:11:44.153 I UDP New client connected from 178.85.134.183:59839
13:11:44.153 I UDP Received session request from 178.85.134.183:59839 with connection ID 1224801937
13:11:44.512 I Chunk New Session: Account ID: 536 Session ID: '6qoq0lgqbm5megf5tmo7o83554'
13:11:44.902 D Chunk control_text='HELLO REVISION=0 MINVER=3151 VER=3186'
13:11:45.245 D Chunk control_text='LOGIN'
13:12:12.170 D Chunk control_text='JOIN'
13:12:57.283 D UDP Filtered out a duplicate packet bunch. Sequence=0x0B00.
13:13:48.138 E Packet Server packet struct (type=2) (client) with opcode 0x000000E1 (225) not found
0000: 00 09 00 CD 02 00 E1 00 - 00 00 00 00 00 00 AE 74 ...............t
13:13:48.138 E Chunk Unhandled opcode 0x000000E1 (225) from 178.85.134.183
13:14:22.520 D Char Saving character 'Sam Thijs' (976)
13:14:22.973 D Char Saved successful for character 'Sam Thijs' (976)
13:14:50.350 E Net Unhandled opcode 0x00000504 (1284) from 178.85.134.183
0000: 00 09 00 09 05 00 04 05 - 00 00 82 01 00 00 01 00 ................
0010: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0020: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0030: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0040: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0050: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0060: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0070: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0080: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0090: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00A0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00B0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00C0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00D0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00E0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00F0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0100: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0110: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0120: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0130: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0140: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0150: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0160: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0170: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0180: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0190: 2B 19 +.
13:15:01.083 E Chunk Unhandled opcode 0x000003D9 (985) from 178.85.134.183
0000: 00 09 02 16 02 00 D9 03 - 00 00 00 00 00 00 20 00 .............. .
13:15:02.206 E Chunk Unhandled movement type in MovementBitStreamData::ReadData : 92 tell a dev!
13:15:59.083 E Net Unhandled opcode 0x00000504 (1284) from 178.85.134.183
0000: 00 09 00 0B 05 00 04 05 - 00 00 82 01 00 00 01 00 ................
0010: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0020: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0030: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0040: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0050: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0060: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0070: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0080: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0090: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00A0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00B0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00C0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00D0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00E0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
00F0: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0100: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0110: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0120: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0130: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0140: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0150: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0160: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0170: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0180: FF FF FF FF FF FF FF FF - FF FF FF FF FF FF FF FF ................
0190: 5F D3 _.
13:16:25.836 D Database Vendor id = 8.[/quote]
^^^^ trainer?

[John Adams]

k Faux, I'm afraid I'm going to have to derail you and ask you to take a look at this as a priority. This is happening way too much lately, and I no longer believe it's not about Training

Crash:

Code: Select all

>	WorldServer.exe!DataStruct::SerializeStringWide(char * buf, int * offset) Line 979	C++
 	WorldServer.exe!DataStruct::Serialize(char * buf, int * offset) Line 739	C++
 	WorldServer.exe!DataStructArray::Serialize(char * buf, int * offset) Line 233	C++
 	WorldServer.exe!DataStruct::Serialize(char * buf, int * offset) Line 745	C++
 	WorldServer.exe!PacketStruct::SerializeIntoBuffer() Line 711	C++
 	WorldServer.exe!SOEProtocolData::SetData(PacketStruct * packet_struct) Line 85	C++
 	WorldServer.exe!PacketStruct::Serialize() Line 724	C++
 	WorldServer.exe!ChunkServer::HandleClientTrainingBegin(const std::shared_ptr<Client> & client) Line 4082	C++
 	WorldServer.exe!ChunkServer::ProcessPackets() Line 411	C++
 	WorldServer.exe!ChunkPacketThread(void * data) Line 142	C++
 	WorldServer.exe!ThreadRun(void * arg) Line 77	C++
 	WorldServer.exe!_callthreadstart() Line 255	C
 	WorldServer.exe!_threadstart(void * ptd) Line 239	C
 	kernel32.dll!7694338a()	Unknown
 	[Frames below may be incorrect and/or missing, no symbols loaded for kernel32.dll]	
 	ntdll.dll!77b49f72()	Unknown
 	ntdll.dll!77b49f45()	Unknown

exact line of code, again:

Code: Select all

client->QueuePacket(out_nontrainable_abilities->Serialize());

Though it does appear to be a different flavor of the same crash, being way down in the Serializer.

Console:
[quote]12:13:48.032 D Char Saving character 'Patootie' (992)
12:13:48.032 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:13:48.032 D Char Saved successful for character 'Patootie' (992)
12:18:48.043 D Char Saving character 'Patootie' (992)
12:18:48.074 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:18:48.074 D Char Saved successful for character 'Patootie' (992)
12:23:48.051 D Char Saving character 'Patootie' (992)
12:23:48.051 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:23:48.051 D Char Saved successful for character 'Patootie' (992)
12:26:58.834 W Chunk Stub behavior for handling OP_ClientAccessObject with race_id ='597'
12:26:58.990 W Chunk Stub behavior for handling OP_ClientAccessObject with race_id ='597'
12:26:59.614 W Chunk Stub behavior for handling OP_ClientAccessObject with race_id ='597'
12:28:48.062 D Char Saving character 'Patootie' (992)
12:28:48.062 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:28:48.077 D Char Saved successful for character 'Patootie' (992)
12:33:48.072 D Char Saving character 'Patootie' (992)
12:33:48.072 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:33:48.087 D Char Saved successful for character 'Patootie' (992)
12:38:48.081 D Char Saving character 'Patootie' (992)
12:38:48.081 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:38:48.081 D Char Saved successful for character 'Patootie' (992)
12:43:48.093 D Char Saving character 'Patootie' (992)
12:43:48.093 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:43:48.093 D Char Saved successful for character 'Patootie' (992)
12:48:48.101 D Char Saving character 'Patootie' (992)
12:48:48.101 D Database WorldDatabase::SaveCharacterInventory: No items needing addition/updating.
12:48:48.117 D Char Saved successful for character 'Patootie' (992)
12:51:53.330 D Vendor ChunkServer::HandleClientMerchantRequest activated.
12:51:53.330 D Vendor ChunkServer::HandleClientRepairMerchant creating vendor.
12:51:53.330 D Char In WorldCharacter::Setting vendor to character Patootie
12:51:58.618 D Item ChunkServer::HandleClientStopShopping activated.
12:51:58.618 D Char In WorldCharacter::Setting vendor to character Patootie
12:52:07.166 D Vendor ChunkServer::HandleClientMerchantRequest activated.
12:52:07.166 D Vendor ChunkServer::HandleClientRepairMerchant creating vendor.
12:52:07.166 D Char In WorldCharacter::Setting vendor to character Patootie
12:52:08.789 D Item ChunkServer::HandleClientStopShopping activated.
12:52:08.789 D Char In WorldCharacter::Setting vendor to character Patootie
12:52:35.900 D Database Vendor id = 8.[/quote]
Again, Vendor id = 8

Thanks

[Faux]

John,
Do you know what chunk this happened in and what class the character is? Vendor_id 8 is a heavy fighter instructor but I want to see if I can recreate the exact same scenario.

[John Adams]

Sorry, I thought I attached the log, but that was the other bug

[quote]12:48:48.101 D Char Saving character 'Patootie' (992)[/quote]
Patootie is class_id 4

[quote]12:12:53.074 D Chunk Starting chunk 'Bordinar's Cleft' (136)[/quote]
That's the chunk

[quote]12:51:53.330 D Vendor ChunkServer::HandleClientMerchantRequest activated.
12:51:53.330 D Vendor ChunkServer::HandleClientRepairMerchant creating vendor.
12:51:53.330 D Char In WorldCharacter::Setting vendor to character Patootie
12:51:58.618 D Item ChunkServer::HandleClientStopShopping activated.[/quote]
This logging needs more information, could use spawn_id, even the name if available there. Type of merchant/vendor/whatever we're calling them this week.

(oh I didn't post the whole log because it was huge)

[zippyzee]

Just to be safe, I commented out one of my lines of code at the end of vendor interaction (HandleClientStopShopping):

Code: Select all

//delete character->GetCurrentVendor(); 
character->SetCurrentVendor(nullptr);

Basically the vendor is created as needed using the vendor class, and the pointer to the vendor stored with the character. When done, the vendor class instance is deleted and the pointer to it removed from the character.

I did experience a predictable crash when I did the vendor delete from within the character class, but once I moved it to the outside in chunkserver.cpp I could not reproduce it, and I tried hard by quickly opening/closing options, switching between vendors, etc.

If I leave it commented out, I don't know how the memory will be cleaned up, if it will automatically or if it will create a slow memory leak. That is not my realm of expertise. I did quite a bit of testing with it with it commented out and there was no noticeable issue over that period of time.

This may not have anything to do with it, but I wanted to eliminate that possibility.

[John Adams]

zippy, thanks for looking into this too, though I'm pretty sure this particular line of code crashed us before any of your vendor work was committed. Searching this forum for that text will reveal the number of times I've posted the crash, so I think you're off the hook (for that one, at least )

[Faux]

I'll try to look at this a bit at lunch today. I hopped on to try and recreate just now, but I need to fix my database up since I'm not getting any npc spawns right now. I recall seeing you post about this after your major db reparsing, John, so I'll go find that thread and patch things up.

The one confusing thing is that the HandleClientTrainingBegin code only uses stack memory. No heap memory is allocated or deleted. I don't see anything wrong with the code yet, but maybe something will present itself when I try testing at lunch. It uses pointers to heap memory when filling the packet array, though, but those pointers are set to nullptr at the end. The only anomaly I see is that the packets are "delete" - ed, but they were never created with new.

Before I remove those delete statements, I want to see if I can recreate the crash though.

[zippyzee]

Code: Select all

ability = abilities_list.GetAbility(result.GetUInt32Str("ability_id"));
			// One more error check because error checking is fun
			if (ability->ability_id < 1)
				LogDebug(LOG_ABILITY, 5, "WorldDatabase::GetTrainableAbilities: Attempted to add invalid Ability id %u to vector.", ability->ability_id);

There is a chance the GetAbility() function will get a nullptr in return. So when you access a ability->ability_id you may get the crash.

I will say that referencing part of a null pointer is 99% of world crashes, in my experience. If nothing else, adding a check there can't hurt:

Code: Select all

[/// Get a pointer to the ability struct from the master list.
			ability = abilities_list.GetAbility(result.GetUInt32Str("ability_id"));
			if (!ability){
				LogError(LOG_ABILITY, 0, "Failed to retrieve ability from abilities list!");
				return;
			}
                        // One more error check because error checking is fun
			if (ability->ability_id < 1)
				LogDebug(LOG_ABILITY, 5, "WorldDatabase::GetTrainableAbilities: Attempted to add invalid Ability id %u to vector.", ability->ability_id);

May not be the cause, but would be helpful in problem-solving anyway. I hope you don't mind me taking a peek. I had a few minutes to spare and am an expert at creating world crashes through my own code techniques.

[Faux]

Good catch Zippy. That could certainly lead to a world crash. I updated the check after that call to GetAbility to prevent it. And don't ever worry about checking over my code. Its what John wants, and its not as though I'm an expert coder or anything .

I've been toying with this and still can't recreate those crashes. I'll keep looking though.

[John Adams]

Server's been down since 3am, guess no one else noticed all day. Sorry.

Haven't seen this issue since Lokked fixed it over a month ago; hope it's not back. With Lokked being AFK, I need someone to make this a priority and look into it. I have no idea how "udp" can be empty in this call, but it was.

Stack

Code: Select all

 	ntdll.dll!77bb8277()	Unknown
 	[Frames below may be incorrect and/or missing, no symbols loaded for ntdll.dll]	
 	WorldServer.exe!Mutex::ReadLock() Line 99	C++
 	WorldServer.exe!UDPServer::GetClient(unsigned int account_id) Line 194	C++
 	WorldServer.exe!ChunkServer::GetClient(unsigned int account_id) Line 288	C++
>	WorldServer.exe!UDPServer::KickDupeClients(unsigned int account_id, unsigned int connection_id) Line 268	C++
 	WorldServer.exe!Net::HandleCharacterSelected(std::shared_ptr<Client> & client, PacketStruct * packet_struct) Line 482	C++
 	WorldServer.exe!Net::Process() Line 161	C++
 	WorldServer.exe!main(int argc, char * * argv) Line 216	C++
 	WorldServer.exe!__tmainCRTStartup() Line 241	C
 	WorldServer.exe!mainCRTStartup() Line 164	C
 	kernel32.dll!7694338a()	Unknown
 	ntdll.dll!77b49f72()	Unknown
 	ntdll.dll!77b49f45()	Unknown

Autos:

Code: Select all

		account_id	60	unsigned int
+		this	0x00000000 {looping=??? process_active=??? shutdown_triggered=??? ...}	ChunkServer *
-		udp	{server_active=??? reading=??? writing=??? ...}	UDPServer
		server_active	<Unable to read memory>	
		reading	<Unable to read memory>	
		writing	<Unable to read memory>	
+		host	0x0000012b <Error reading characters of string.>	char[256]
+		port	0x0000022b <Error reading characters of string.>	char[8]
+		socket	{sock=??? address={addr={sa_family=??? sa_data=0x0000023a <Error reading characters of string.> } len=...} ...}	Socket
+		clients	{ size=??? }	std::map<unsigned int,std::shared_ptr<Client>,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,std::shared_ptr<Client> > > >
+		disconnecting_clients	{ size=??? }	std::map<unsigned int,DisconnectingClient *,std::less<unsigned int>,std::allocator<std::pair<unsigned int const ,DisconnectingClient *> > >
+		incoming	{ size=??? }	std::deque<ClientPacketData *,std::allocator<ClientPacketData *> >
+		m_clients	{name=0x00000370 <Error reading characters of string.> lock={Ptr=??? } }	Mutex
+		m_disconnecting_clients	{name=0x00000394 <Error reading characters of string.> lock={Ptr=??? } }	Mutex
+		m_incoming	{name=0x000003b8 <Error reading characters of string.> lock={Ptr=??? } }	Mutex
		read_timeout	<Unable to read memory>	
+		read_list		std::forward_list<SocketData *,std::allocator<SocketData *> >

Log: Note the server ran fine for days, so this is just what happened today (the 27th)

[The extension log has been deactivated and can no longer be displayed.]