Skip to main content

Hey All,

I'm trying to create a python script which can be called from a BP program (use PyCallFunction or the like) which will log to fluentd.

I have something which works on the first run of the BP program, but crashes on the second call.

To replicate you don't need the BP program, the following will do, from TCL

```

PYTHON

from fluent.sender import FluentSender

sender.emit('uv', { 'level': 'info', 'message': "hello"})

sender.emit('uv', { 'level': 'info', 'message': "world!"})

quit()

PYTHON

from fluent.sender import FluentSender

sender.emit('uv', { 'level': 'info', 'message': "hello"})

```

The error is

```

  File "/home/u2dev/lib64/python3.11/site-packages/msgpack/__init__.py", line 36, in packb
    return Packer(**kwargs).pack(o)
           ^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

```

It looks like on the second go round msgpack can't import Packer for a .so file.

I've tried setting REINIT.PYTHON OFF, but no luck.

This is on UV 11.4.1 and my python requirements.txt file is contains

fluent-logger==0.11.1
msgpack==1.1.0

I'm hoping one of the bright sparks here can help.

Thanks

Gary



------------------------------
Gary Miller
Helix Collective
------------------------------

Hey All,

I'm trying to create a python script which can be called from a BP program (use PyCallFunction or the like) which will log to fluentd.

I have something which works on the first run of the BP program, but crashes on the second call.

To replicate you don't need the BP program, the following will do, from TCL

```

PYTHON

from fluent.sender import FluentSender

sender.emit('uv', { 'level': 'info', 'message': "hello"})

sender.emit('uv', { 'level': 'info', 'message': "world!"})

quit()

PYTHON

from fluent.sender import FluentSender

sender.emit('uv', { 'level': 'info', 'message': "hello"})

```

The error is

```

  File "/home/u2dev/lib64/python3.11/site-packages/msgpack/__init__.py", line 36, in packb
    return Packer(**kwargs).pack(o)
           ^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not callable

```

It looks like on the second go round msgpack can't import Packer for a .so file.

I've tried setting REINIT.PYTHON OFF, but no luck.

This is on UV 11.4.1 and my python requirements.txt file is contains

fluent-logger==0.11.1
msgpack==1.1.0

I'm hoping one of the bright sparks here can help.

Thanks

Gary



------------------------------
Gary Miller
Helix Collective
------------------------------

Gary,

I don't know whether this helps, but I found this reference:

msgpack is JSON-like format.
Like JSON, msgpack is not support directly serializing any custom object.
If you want it, you should use pickle.

Regards

JJ



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Gary,

I don't know whether this helps, but I found this reference:

msgpack is JSON-like format.
Like JSON, msgpack is not support directly serializing any custom object.
If you want it, you should use pickle.

Regards

JJ



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

I am using the msgpack library indirectly, via fluent-logger, so I don't have much choice to pick a serialization option.



------------------------------
Gary Miller
Self Registered
Redfern AU
------------------------------

I am using the msgpack library indirectly, via fluent-logger, so I don't have much choice to pick a serialization option.



------------------------------
Gary Miller
Self Registered
Redfern AU
------------------------------

Gary,

If msgpack is indeed the root vause and it's not possible to use an alternative, have you thought about launching a PHANTOM to launch the program?

I've seen references to msgpickle as well which supports custom serialisation, but my Python is not up to snuff in this area.

Regards

JJ 



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

Gary,

If msgpack is indeed the root vause and it's not possible to use an alternative, have you thought about launching a PHANTOM to launch the program?

I've seen references to msgpickle as well which supports custom serialisation, but my Python is not up to snuff in this area.

Regards

JJ 



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------

John,

Thanks for your reply.

Can you be more specific about what you mean by a PHANTOM?

The problem I bumped into seams to be a generic u2py problem.

That is it error on run on a import for the second time, where the import is coming from a .so file.

I tried a different approach, instead of using fluent-logger and therefore msgpack, I use a async post request.

Using the aiohttp caused the same error as msgpack, when swapping to the httpx library it worked.

Do anyone have an idea why the following would fail?

"from ._quoting import _Quoter, _Unquoter"

"from ._cmgpack import Packer"

Both imports seems to be from .so files.



------------------------------
Gary Miller
Self Registered
Redfern AU
------------------------------

John,

Thanks for your reply.

Can you be more specific about what you mean by a PHANTOM?

The problem I bumped into seams to be a generic u2py problem.

That is it error on run on a import for the second time, where the import is coming from a .so file.

I tried a different approach, instead of using fluent-logger and therefore msgpack, I use a async post request.

Using the aiohttp caused the same error as msgpack, when swapping to the httpx library it worked.

Do anyone have an idea why the following would fail?

"from ._quoting import _Quoter, _Unquoter"

"from ._cmgpack import Packer"

Both imports seems to be from .so files.



------------------------------
Gary Miller
Self Registered
Redfern AU
------------------------------

Gary,

Regarding PHANTOMs, I was suggesting that the application code that accessed Python and msgpack - even if indirectly - could be fired off as a Phantom on a 'fire once and forget' basis each time it was needed. In that way my thought was each PHANTOM would only make the Python call once before termination, thus avoiding any attempt at any re-entry.

While on the face of it REINIT-PYTHON looks like it might resolve the serialisation but I understand not, and the command does come with a caveat:

  • Note: Depending on the coding techniques and assumptions made in the Python code being used,
    the implied re-initialization of the Python environment might not be supported when used for a
    second time in the same session.

As regards Python specifics on this library - I've been gleaning Python forums and will have to leave that to more up-to-date Python specialists (it's been a while for me since I retired).

Regards

JJ



------------------------------
John Jenkins
Thame, Oxfordshire
------------------------------