Python Twisted TCP application - How to prevent incoming message loss by
blocking process
I have 10 messages/second(total activity) coming in on TCP from 40
clients. I need to take each message and do a 5 second process (look up a
webservice, do some DB queries and finally write the results to the DB).
How do I separate messages coming in from the slow 5 second process? Also
I might receive another message from a client while already processing a
message for that client. I NEVER want to lose a message.
I'm experimenting with ZeroMQ. I'm sending the message to a queue. I've
got another script that picks up the messages from ZeroMQ. So far so good.
What can I do after picking up the message to allow parallel processing of
the messages? (Threads?, Twisted callbacks?, Multiprocessing?). I don't
have to return a response to the client.