mORMot Open Source

Wawi is a short for Warenwirtschaft.
JTL-Software-GmbH are Germans. For such Wawi is a fundamental word in their business language (just like beer for English -> beer -> bearer)

https://mailman.nginx.org/pipermail/ngi … 08252.html

An alternative of TSynUniqueIdentifierGenerator.ComputeNew is using of sequences to compute the ID's before building the Batch.

Hi Arnaud,

I hope you are back from holidays...

The issue remains with current mORMot2 under -O2/-O3. With -O1 it works. Do you plan some fixing here? Could you generate the vm for FreeBSD?

Daniel

Sorry, but -02/-03 doesn't work with current version under fpc 3.2.0.
Disable inlining of FastNewString seems to be not enough:

Thread 4 "http1root" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6ae5700 (LWP 32094)]
ADDTEXT (this=0x0, TEXT=0x0, ESCAPE=TWJSONESCAPE)
    at ../../mORMot2/src/core/mormot.core.json.pas:6776
6776        AddDirect('"');

Note: fpc 3.2.2 (and my FreeBSD) all works without issue under current mORMot2 version.

Yes I know.

In test.net.proto.pas there is an issue starting from line

1543   try^M
   1544     // validate raw TCP tunnelling^M

But I'm lost with all of that.

fpc -iW shows

3.2.0-r45643

Memleak reported comes from TTestCoreEcc

disable inlining in FastNewString was it! Thanks a lot.

(mormot2tests doesn't compile, I don't know what's wrong. No error, just Error in ./build_fpc.sh on line 117)

Thanks for explain, make sense.

So when using plain THttpServer, what's your proposal to set ServerKeepAliveTimeOut:= 0 (disabled keep alive)?

I'm creating just a TRestHttpServer which could creates different kinds of HttpServers (Socket, Api, Async) and ServerKeepAliveTimeOut property isn't public over  TRestHttpServer.HttpServer.

--> THttpServer(aTRestHttpServerInstance).ServerKeepAliveTimeOut:= 0; <-- looks like a hack

@dcoun
As I remember I had the same issue with TSynLog for years. It's probably not related to mORMo2 but Windows:

https://stackoverflow.com/questions/687 … owsservice

Sure is a great word and what does it mean to you, when I am?
Better I show you what the log file says:

20250211 15375400  !  +            uHTTPServer.TSOneHttpServer(020f4f28).Create useHttpAsync (secNone) on port 8281
20250211 15375400  !  +                mormot.net.async.THttpAsyncConnections(03205f68).Create(THttpAsyncServerConnection,root,32)
20250211 15375401  " info  SetThreadName 7f69b8b6c640 140092047279680=R0:root
20250211 15375401  " trace uRestServerDB.TSOneRestServerDB(01dfe208) BeginCurrentThread(TAsyncConnectionsThread) root=root ThreadID=7f69b8b6c640 'R0:root' ThreadCount=1

Hello Arnaud,

I could find the location of our culprit today.
But I'm not able to understand it by now and have to test more.

Thanks
Daniel

@ab
Today we had "same" behavior like @itSDS (I'm looking for errors on our side first and since weeks), but it's not a client issue on our side.
1. service isn't responsive any more (/timestamp over curl gives timeout too).
2. absolut no cpu-usage in any thread
3. no exceptions or error in (verbose) log file before.
4. absolut no timeout in postgres.

Our service runs since yesterday evening and we had about 40' request per hour till the issue at 2:20 pm.
We are using mORMot commit #fe67be7b8 / heap.inc / linux (fpc 3.2.0).

I don't want to waste your time. It's a fpc issue I don't understand.

locale:
LANG=C.UTF-8
LC_CTYPE="C.UTF-8"
LC_COLLATE="C.UTF-8"
LC_TIME="C.UTF-8"
LC_NUMERIC="C.UTF-8"
LC_MONETARY="C.UTF-8"
LC_MESSAGES="C.UTF-8"
LC_ALL=

fpc:
compiled with -FcUTF-8
mormot.defines.inc inlcuded

file SOneSrv2.dpr:
Unicode text, UTF-8 (with BOM) text (also tested without BOM)

in .dpr just a writeln('ü') -> output: '?'

Thanks you,
Daniel

all my units have a options.inc include with your
{$I mormot.defines.inc}

Just a question from today when working under linux (system codepage utf8), unit.pas is set to utf8 using fpc 3.2.2:

When I make a function Example(const s: RawJSON/RawUtf8/UTF8String) call with:

Example('äöü') -> it's not working any more. I have to call
Example(StringToUtf8('äöü')) to get the right string.

Is this a new behavior from this update. I can't remember to have such issues in past.

I don't have a stack trace on my production system, just a log file (in verbose logging).

The mongdb call is a simple aggregate with a small result, called over a sicSingle service method. Not complex, just a very simple method.

I believe the root of the issue comes from our side, not from mORMot2. But I don't find any entries of interest in my logs.
The mongodb call methods are so simple, that there couldn't be any issue come from. I have 32 workers and every worker has its own mongodb connection in his own threadvar. No race condition, no blocking issue.
MongoDB calls are the main work in our service. If any worker connection would have an issue I would have a crash or errors instantly.

Atm I don't use your new rewrite version (just ef8764e2c).

To find out if async server is involved, should I switch to useHttpSocket and watch?

cmem

@ab Yes mem issue is outside of pipeline.
Today we had such an issue in our program (of course without pipeline):

SOneSrv2: malloc.c:4302: _int_malloc: Assertion `(unsigned long) (size) >= (unsigned long) (nb)' failed

We had no error in Postgres log at all. But we are using cmem.

Threadvar's are created in a special memory area (thread-local storage). At compile time the addresses of that space for a thread created at runtime isn't fix.
Therefore libraries linked at compile time (which packages are) could have an issue with threadvar's. Dynamically loaded libraries are fine, because they will loaded at runtime, when the threads run.

Because of this it doesn't matter whether a threadvar is declared in interface or implementation part.

I've implemented this task some time ago to check if I could handle it. It's not so easy, but perhaps the situation is more easy if I would have a real AD.

To verify the JWT you need the public key.
The public key in pem format you get from the x5c value. In my test case (at microsoft) the key changes every time (I can't remember the interval e.g. every 30 days).
To get it you have to call -> https://login.microsoftonline.com/'+ response.tenantId +'/discovery/v2.0/keys?appid=...
and compare the kid included in the JWT which you want to verify.

If you have the public key you can verify the JWT with this func:

function VerifyJWT(const aJWT: RawByteString; const aPublicKey: RawByteString; out JWT: Variant): Boolean;
  function VerifyWithmORMot(const aJWT: RawByteString; out JWT: Variant): Boolean;
  var
    Header, Payload, Signature: RawByteString;
    sigBinary, msgBinary: TBytes;
    idx, step: Integer;
    aBuf: PAnsiChar;
  begin
    Header:= '';
    Payload:= '';
    Signature:= '';
    idx:= 0;
    step:= 0;
    aBuf:= PAnsiChar(aJWT);
    while aBuf[idx] <> '' do begin
       INC(idx);
       if aBuf[idx] = '.' then begin
         INC(step);
         if step=1 then
           SetString(Header, aBuf, idx)
         else
         if step=2 then
           SetString(Payload, aBuf, idx)
         else
           SetString(Signature, aBuf, idx);
         INC(aBuf, idx+1);
         idx:= 0;

       end;
    end;

    if (idx > 0) and (step < 3) then
      SetString(Signature, aBuf, idx);

    JWT:= _Obj([]);
    sigBinary:= Base64UrlDecode(Signature);
    RawByteStringToBytes(Header+'.'+Payload, msgBinary);

    Result:= OpenSslVerify('', '', Pointer(msgBinary), Pointer(aPublicKey), Pointer(sigBinary), Length(msgBinary), Length(aPublicKey), Length(sigBinary));
    if Result then
      JWT:= Js2Var(Base64uriToBin(PAnsiChar(Payload), Length(Payload)));
  end;

begin
  Result:= VerifyWithmORMot(aJWT, JWT);
end;

I wrote again because I think it could be useful to stabilize the interfaces/network (mormot.core.interfaces / mormot.core.soa) part of the framework, so it is useful for all users:

My pov is, executing of interfaced methods isn't thread safe in some circumstances. Two parallel (sicSingle) service calls comes and their params will be mixed/override. In my case there are such corrupt bison's in mongodb queries.

@Ab could you please check TInterfaceMethodExecuteCached if there is anything what could go wrong under heaver load? All my wrk tests looks good but the live system has this error behavior and all other processes looks good, also the syslog looks good so far.

I've commented out this part of TInterfaceMethodExecuteCached .Aquire

if fCached.TryLock then
  begin
    // reuse this shared instance between calls
    SetOptions(opt);
    exec := self;
    fCachedWR.CancelAllAsNew;
    WR := fCachedWR;
  end

but its not enough or not the right place.
Are there any other changes in past that comes with https://blog.synopse.info/?post/2022/01 … e-Them-All the that I could test easily?

Perhaps it is not an issue of wrong locking but of not initialization of something reused because of caching?

Hi ab,

just fyi the error in my service from today and the syslog entries bring me to the nested option for lxc containers and I think I'll give it a try:

https://forum.proxmox.com/threads/lxc-a … -13.36173/