Compatibility of WaitSet and wait operations with threads in python #1796

Note: Moved from GitHub discussion to new iceoryx2 community

Hello together, I am currently using iceoryx2 with the python language bindings and aim to develop a simple server/client exchange via the request/response pattern.

To this end, I normally use a Server class that sets up communication with iceoryx2 and then starts a thread to run the event-based communication. However, I ran into the issue that after setting up this thread, the Server class does not initialize properly.
I have isolated all of this in, what I think, is the most minimal example for this behaviour.

The server event_loop sets up the iox2 objects and then performs wait_and_process() via the waitset to listen to any events.
The client function also sets up the iox2 objects and directly sends the payload, notifies the request event service and waits for the response from the eventloop.

However, executing this code does not terminate. To rule out that this is due wait_and_process() being activated, thus needing an event to wake_up and terminate, I replaced the ‘while True’ loop with a ‘running’ variable that ensures the loop is only run once.
Still, just by looking at the terminal, execution seems to freeze once the event_loop ist started.

Some more things I already considered.

  1. Running the event_loop and the client_fn in separate processes works, also with a while True: loop
  2. Simply replacing the wait_and_process function with wait_and_process_with_timeout() results in a timeout on the client side

I am a bit clueless on what to do now. I also suspected whether the GIL is handled correctly, but the iceoryx2 documentation specifically states that for blocking_waits in python the GIL is released.

My question is therefore, does anyone know what else could be the issue with this code and can confirm that this form of communication properly aligns with the iceoryx2 communication patterns/examples?

minimal_issue_example.py
import logging
import time
import threading
import iceoryx2 as iox2
import ctypes

main_logger = logging.getLogger(__name__)
cycle_time = iox2.Duration.from_millis(1)

SERVICE_NAME = "Test"

class TransmissionData(ctypes.Structure):
    """The strongly typed payload type."""

    _fields_ = [("x", ctypes.c_int32)]

    def __str__(self) -> str:
        """Returns human-readable string of the contents."""
        return f"TransmissionData {{ x: {self.x} }}"

    @staticmethod
    def type_name() -> str:
        """Returns the system-wide unique type name required for communication."""
        return "TransmissionData"


def handle_event(server, listener, notifier, _logger):
    """
    Process all pending events and requests.
    """
    for event in listener.try_wait_all():
        _logger.info(event)
        if not event:
            break

        active_request = server.receive()
        if not active_request:
            _logger.warning("Event signaled but no request available in queue")
            continue
        _logger.info("Request Received")

        payload: TransmissionData = active_request.payload().contents
        old_x = payload.x

        response_payload = TransmissionData()
        response_payload.x = 2 * old_x

        # Send response
        active_request.send_copy(response_payload)
        notifier.notify()
        _logger.info("Response Sent")

        active_request.delete()


def event_loop():
    # Setup of Iceoryx2 objects
    event_logger = logging.getLogger(__name__ + "_event_loop")
    node = iox2.NodeBuilder.new().create(iox2.ServiceType.Ipc)
    service_rr = (
        node.service_builder(iox2.ServiceName.new(SERVICE_NAME))
        .request_response(TransmissionData, TransmissionData)
        .open_or_create()
    )

    request_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Request")).event().open_or_create()
    response_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Response")).event().open_or_create()

    iox_server = service_rr.server_builder().create()
    listener = request_event.listener_builder().create()
    notifier = response_event.notifier_builder().create()

    waitset = iox2.WaitSetBuilder.new().create(iox2.ServiceType.Ipc)
    guard = waitset.attach_notification(listener)

    # Handling eventloop via iceoryx2
    running = True
    while running:
        attachments, result = waitset.wait_and_process()
        # attachments, result = waitset.wait_and_process_with_timeout(cycle_time)
        if result in (
            iox2.WaitSetRunResult.TerminationRequest,
            iox2.WaitSetRunResult.Interrupt,
        ):
            break

        for attachment in attachments:
            if attachment == iox2.WaitSetAttachmentId.from_guard(guard):
                event_logger.info("Event Handle Triggered")
                handle_event(iox_server, listener, notifier, event_logger)
        running = False

    return


def client_fn():
    # Setup of Iceoryx2 objects
    client_logger = logging.getLogger(__name__ + "_client_fn")

    node = iox2.NodeBuilder.new().create(iox2.ServiceType.Ipc)
    service_rr = (
        node.service_builder(iox2.ServiceName.new(SERVICE_NAME))
        .request_response(TransmissionData, TransmissionData)
        .open_or_create()
    )
    request_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Request")).event().open_or_create()
    response_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Response")).event().open_or_create()

    client = service_rr.client_builder().create()
    notifier = request_event.notifier_builder().create()
    listener = response_event.listener_builder().create()

    # Sending data via iceoryx2 request/response
    payload = TransmissionData()
    payload.x = 5

    iox_request = client.loan_uninit()
    iox_request = iox_request.write_payload(payload)
    pending_response = iox_request.send()
    notifier.notify()
    client_logger.info(f"Request sent.")

    wait_result = listener.timed_wait_one(iox2.Duration.from_secs(1))
    if not wait_result:
        client_logger.error(f"Response timeout for service: {SERVICE_NAME}")

    iox_response = pending_response.receive()
    if iox_response:
        response: TransmissionData = iox_response.payload().contents
        client_logger.info(f"Response received with value {str(response)}")

    return


if __name__ == "__main__":
    logging.basicConfig(level=logging.DEBUG)

    event_thread = threading.Thread(
        target=event_loop,
        name="EventLoop",
        daemon=False,
    )
    client_thread = threading.Thread(
        target=client_fn,
        name="Client",
        daemon=False,
    )

    event_thread.start()
    main_logger.info("Event Thread Started")

    client_thread.start()
    main_logger.info("Client Thread Started")

This is weird and could be an iceoryx2 bug. Thanks for providing the example! If this is indeed a bug I would create an issue on github and keep you posted!

@TDietz21 I tested your example with the current main branch on iceoryx2. This includes a major event refactoring that added stability and guarantees that events are never lost.

Therefore, I had to rename the function try_wait_all and timed_wait_one to try_wait and timed_wait. Those functions now always return a vector with all received events and the count, how often they were triggered.

Your code snippet produced this output repeatedly:


WARNING:“Config.global_config()”:No config file was loaded, a config with default values will be used.
INFO:main:Event Thread Started
INFO:main:Client Thread Started
INFO:__main___event_loop:Event Handle Triggered
INFO:__main___event_loop:EventActivation { id: EventId(0), count: 1 }
INFO:__main___event_loop:Request Received
INFO:__main___event_loop:Response Sent
INFO:__main___client_fn:Request sent.
INFO:__main___client_fn:Response received with value TransmissionData { x: 10 }
```

Is this what you expected to see?

The issues I mentioned occurred with the iceoryx2 version 0.8.1.
Updating to 0.9.3 solved the issue with the blocking thread for me.
I also tried the current state on the main branch and after renaming the functions you mentioned, it worked the same.

This was the code I ended up using:

updated_minimal_example.py
import logging
import time
import threading
import iceoryx2 as iox2
import ctypes

main_logger = logging.getLogger(__name__)

SERVICE_NAME = "Test"
RUNNING = True

class TransmissionData(ctypes.Structure):
    """The strongly typed payload type."""

    _fields_ = [("x", ctypes.c_int32)]

    def __str__(self) -> str:
        """Returns human-readable string of the contents."""
        return f"TransmissionData {{ x: {self.x} }}"

    @staticmethod
    def type_name() -> str:
        """Returns the system-wide unique type name required for communication."""
        return "TransmissionData"


def handle_event(server, listener, notifier, _logger):
    """
    Process all pending events and requests.
    """
    for event in listener.try_wait_all():
        _logger.info(event)
        if not event:
            break

        active_request = server.receive()
        if not active_request:
            _logger.warning("Event signaled but no request available in queue")
            continue
        _logger.info("Request Received")

        payload: TransmissionData = active_request.payload().contents
        old_x = payload.x

        response_payload = TransmissionData()
        response_payload.x = 2 * old_x

        # Send response
        active_request.send_copy(response_payload)
        notifier.notify()
        _logger.info("Response Sent")

        active_request.delete()


def event_loop(node):
    # Setup of Iceoryx2 objects
    event_logger = logging.getLogger(__name__ + "_event_loop")
    
    service_rr = (
        node.service_builder(iox2.ServiceName.new(SERVICE_NAME))
        .request_response(TransmissionData, TransmissionData)
        .open_or_create()
    )

    request_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Request")).event().open_or_create()
    response_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Response")).event().open_or_create()

    iox_server = service_rr.server_builder().create()
    listener = request_event.listener_builder().create()
    notifier = response_event.notifier_builder().create()

    waitset = iox2.WaitSetBuilder.new().create(iox2.ServiceType.Ipc)
    guard = waitset.attach_notification(listener)

    # Handling eventloop via iceoryx2
    global RUNNING
    while RUNNING:
        attachments, result = waitset.wait_and_process()
        if result in (
            iox2.WaitSetRunResult.TerminationRequest,
            iox2.WaitSetRunResult.Interrupt,
        ):
            break

        for attachment in attachments:
            if attachment == iox2.WaitSetAttachmentId.from_guard(guard):
                event_logger.info("Event Handle Triggered")
                handle_event(iox_server, listener, notifier, event_logger)

    return


def client_fn(node):
    # Setup of Iceoryx2 objects
    client_logger = logging.getLogger(__name__ + "_client_fn")

    service_rr = (
        node.service_builder(iox2.ServiceName.new(SERVICE_NAME))
        .request_response(TransmissionData, TransmissionData)
        .open_or_create()
    )
    request_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Request")).event().open_or_create()
    response_event = node.service_builder(iox2.ServiceName.new(SERVICE_NAME + "/Response")).event().open_or_create()

    client = service_rr.client_builder().create()
    notifier = request_event.notifier_builder().create()
    listener = response_event.listener_builder().create()

    for i in range(10):
        # Sending data via iceoryx2 request/response
        payload = TransmissionData()
        payload.x = i * 5

        iox_request = client.loan_uninit()
        iox_request = iox_request.write_payload(payload)
        pending_response = iox_request.send()
        notifier.notify()
        client_logger.info(f"Request sent.")

        wait_result = listener.timed_wait_one(iox2.Duration.from_secs(1))
        if not wait_result:
            client_logger.error(f"Response timeout for service: {SERVICE_NAME}")

        iox_response = pending_response.receive()
        if iox_response:
            response: TransmissionData = iox_response.payload().contents
            client_logger.info(f"Response received with value {str(response)}")

    return


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)

    client_node = iox2.NodeBuilder.new().create(iox2.ServiceType.Ipc)
    server_node = iox2.NodeBuilder.new().create(iox2.ServiceType.Ipc)

    event_thread = threading.Thread(
        target=event_loop,
        name="EventLoop",
        args=[server_node],
        daemon=False,
    )
    client_thread = threading.Thread(
        target=client_fn,
        name="Client",
        args=[client_node],
        daemon=False,
    )

    event_thread.start()
    main_logger.info("Event Thread Started")

    client_thread.start()
    main_logger.info("Client Thread Started")

    time.sleep(1)

    print("Done")

Maybe an interesting change I made: when trying my original code on v0.9.3, I kept getting `WARNING:SharedNodeState{…}: Unable to remove node resources.`
Always at the end of each run when the threads where terminated.

I believe this was due to the iox2::Node being initialized within each thread, so when the thread terminated, the iox2::Node went out of scope but might not have been cleaned up properly as the surrounding process was still running.

Anyway, I managed to avoid the Warning by moving the node definition to the main thread such that the nodes go out of scope simultaneously with the process terminating

Thank you very much for your help!