mORMot Open Source

DonAlfredo

Member

Registered: 2023-10-31

Posts: 16

DocVariant to Object and back conversion problem

Hello.
The following code does not work for RawBlobs.
Product is a CollectionItem with a RawBlob as one of its fields.

rA := Rtti.RegisterClass(PClass(Product)^);
ObjectToVariant(Product,TD2,[woRawBlobAsBase64]);
DocVariantToObject(_Safe(TD2)^,Product,rA);

After this conversion, the Rawblob field contains the Base64 encoded data and not the original contents.
Am I missing something ?

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: DocVariant to Object and back conversion problem

Hello.

It is indeed not supported, because
1) mormot.core.rtti doesn't include mormot.core.buffers so can't check for the Base64-encoding at this level
2) the woRawBlobAsBase64 format has no magic trailer, so it is impossible to know for sure that it needs to be decoded or not - it is possible that a valid BASE-64 text is in fact a binary...

Is woRawBlobAsBase64 really needed?
It seems just easier to me to use a RawUtf8 with manual Base-64 encoding instead.

DonAlfredo

Member

Registered: 2023-10-31

Posts: 16

Re: DocVariant to Object and back conversion problem

Is woRawBlobAsBase64 really needed?

I am sending over some fields by converting them to a variant like this

rA := Rtti.RegisterClass(PClass(Product)^);
pa := rA.Props.Find(FieldName);
pa^.GetValueVariant(Product,TVarData(TD2), @JSON_[mDefault]);
TDocVariantData(TD).AddValue(FieldName,TD2);

Problem with the thumb: binary data containing zero's. So, when sending over (interface based), the zero's prevent correct json data when the variant is converted into json and back. The [json] data gets truncated at the first zero encountered.
Naturally, like rawblobs, using base64 solved this. As done by ObjectToVariant. But no auto-detection of base64 contents when converting back.

it is possible that a valid BASE-64 text is in fact a binary

Indeed !

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: DocVariant to Object and back conversion problem

It seems just easier to me to use a RawUtf8 with manual Base-64 encoding instead.

DonAlfredo

Member

Registered: 2023-10-31

Posts: 16

Re: DocVariant to Object and back conversion problem

I was hoping for something (some magic) I missed. But I will use manual detection on client and server as you suggest.
Thanks Ab.

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: DocVariant to Object and back conversion problem

Wait I will do something better.

Can you try with woRawByteStringAsBase64Magic ?
https://github.com/synopse/mORMot2/commit/e384850d0

It would add a "magic" to the output, which should now be recognized as BASE-64 encoding with:
https://github.com/synopse/mORMot2/commit/86a81f3c5

DonAlfredo

Member

Registered: 2023-10-31

Posts: 16

Re: DocVariant to Object and back conversion problem

I am sorry, but I cannot get it to work unfortunately. I am doing something wrong, but have no clue.
Code.

TProduct2 = class(TCollectionItem)
strict private
    fProductCode      : RawUTF8;
    fThumb            : RawBlob;
published
    property ProductCode : RawUTF8 read fProductCode write fProductCode;
    property Thumb       : RawBlob read fThumb write fThumb;
end;

Product2:=TProduct2.Create(nil);
Product2.ProductCode:='yolo';
Product2.Thumb:='don'#0'alfredo';
//TD2:=_ObjFast(Product2);
//TD2:=_ObjFast(Product2,[woRawBlobAsBase64]);
TD2:=_ObjFast(Product2,[woRawBlobAsBase64,woRawByteStringAsBase64Magic]);
json:=TDocVariantData(TD2).ToJson; // as check of raw contents
DocVariantToObject(_Safe(TD2)^,Product2);

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: DocVariant to Object and back conversion problem

woRawByteStringAsBase64Magic is exclusive of woRawBlobAsBase64 I guess.

Try

TD2:=_ObjFast(Product2,[woRawByteStringAsBase64Magic]);

DonAlfredo

Member

Registered: 2023-10-31

Posts: 16

Re: DocVariant to Object and back conversion problem

TD2:=_ObjFast(Product2,[woRawByteStringAsBase64Magic]);
json:=TDocVariantData(TD2).ToJson;

Result: json = 'null'
Very strange !

ab

Administrator

From: France

Registered: 2010-06-21

Posts: 15,005

Website

Re: DocVariant to Object and back conversion problem

Try to define your Thumb field as RawByteString.
RawBlob was meant for the ORM, for which null is expected by default, unless BLOBs are told to be explicitly retrieved from the DB (which is usually slower).
RawByteString should not suffer from this behavior.
- also ensure that you have the latest sources with proper _JS_RawByteString().

Or just don't use any in-between variant, but RawJson and direct JSON serialization.
When transmitted over interface services, RawJson would be safer and faster, since it would bypass all TDocVariant fields allocation.