mORMot Open Source

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 50

RTTI and NaN values

Recently, I started to see a problem when using 'DynArrayLoadJson' in an application that had worked well before, but was beginning to fail. I checked the (huge, more than 100Mb) text file generated from a previous 'DynArraySaveJson' with that 'JSON', trying to realize what could be causing that 'nil' response from 'DynArrayLoadJson'. After some time, I found that some 'Double' fields had recently been added to the huge record and in their processing they had been calculated, but because of a 'bug', sometimes a division by zero was happening in those fields, what was leading those 'Double' fields written as 'NaN' and making 'DynArrayLoadJson' fail and returning 'nil'.

After that, I reversed the 'No Exception' behavior of Delphi 12 to the previous versions behavior using 'System.Math.SetExceptionMask( [exPrecision, exUnderflow, exDenormalized])', because 'division by zero' errors may happen to anyone once in a while, and the latest Delphi version change of that behavior was leaving programmers kind of blind to those eventual errors.

My question is: Is it possible to add a field on 'DynArrayLoadJson' that could return a pair like 'field_a="a"' for the first error found showing which field raised the conversion problem when 'DynArrayLoadJson' returned 'nil'?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: RTTI and NaN values

I would not add any such field to DynArrayLoadJson(), because it may be really confusing.
This function is already complex enough.

We rather find out what exactly is wrong, and fix it.

Where is "Nan" written? I guess from FloatToJsonNan() after a wrong conversion to double. Is it really stored as "Nan" in the JSON?

Then we may now need to identify such JSON_NAN[] values from text.
I have just made some commits to handle those values from JSON.
See https://github.com/synopse/mORMot2/commit/be50accb8
- but not on Delphi Win32 currently

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 50

Re: RTTI and NaN values

The array of records saved with 'DynArraySaveJson' has several 'Double' fields that are calculated, and only a few of them resulted in 'NaN' because of the 'division by zero' error. This was already fixed. At that occasion, Delphi wrote those sparse 'NaN' values to those fields, and 'DynArraySaveJson' saved those 'Double' fields with error as "field_a":"NaN". Yes, they had been stored in the 'JSON' file exactly this way. When I needed to load the 'JSON' file in other opportunities, 'DynArrayLoadJson' was returning 'nil'.

It was really hard to find the cause of the conversion error because of the new way Delphi is looking at some exceptions, but as I told you above, I reversed it. However, is there some way to be informed which error(s) caused a 'nil' return value from 'DynArrayLoadJson', even if not through a new 'out field'? When one is dealing with a huge amount of data, it might be complicated to find out what could have caused the problem.

Thanks for your help!

Last edited by Prometeus (2025-05-09 13:32:00)

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: RTTI and NaN values

I have added "NaN" parsing support in the asm of GetExtended() for Delphi Win32 too.
(this x87 FPU asm is just a pain in the ...)

Prometeus

Member

From: USA

Registered: 2020-11-20

Posts: 50

Re: RTTI and NaN values

Ok, great!

I really appreciate your support!