Twisted的延时处理机制
来源:互联网 发布:小米air 装linux 编辑:程序博客网 时间:2024/06/13 15:29
Trac back: http://twistedmatrix.com/projects/core/documentation/howto/defer.html
Deferred Reference
- Callbacks
- Multiple callbacks
- Visual Explanation
- Errbacks
- Unhandled Errors
- Handling either synchronous or asynchronous results
- Handling possible Deferreds in the library code
- DeferredList
- Other behaviours
- Class Overview
- Basic Callback Functions
- Chaining Deferreds
- See also
This document is a guide to the behaviour of the twisted.internet.defer.Deferred
object, and to variousways you can use them when they are returned by functions.
This document assumes that you are familiar with the basic principle thatthe Twisted framework is structured around: asynchronous, callback-basedprogramming, where instead of having blocking code in your program or usingthreads to run blocking code, you have functions that return immediately andthen begin a callback chain when data is available.
See these documents for more information:
- Asynchronous Programming with Twisted
After reading this document, the reader should expect to be able todeal with most simple APIs in Twisted and Twisted-using code thatreturn Deferreds.
- what sorts of things you can do when you get a Deferred from afunction call; and
- how you can write your code to robustly handle errors in Deferredcode.
Unless you're already very familiar with asynchronous programming,it's strongly recommended you readthe Deferreds section of theAsynchronous programming document to get an idea of why Deferredsexist.
Callbacks
A twisted.internet.defer.Deferred
is a promise thata function will at some point have a result. We can attach callback functionsto a Deferred, and once it gets a result these callbacks will be called. Inaddition Deferreds allow the developer to register a callback for an error,with the default behavior of logging the error. The deferred mechanism standardizes the application programmer's interface with all sorts of blocking or delayed operations.
from twisted.internet import reactor, defer
def getDummyData(x):
"""
This function is a dummy which simulates a delayed result and
returns a Deferred which will fire with that result. Don't try too
hard to understand this.
"""
d = defer.Deferred()
# simulate a delayed result by asking the reactor to fire the
# Deferred in 2 seconds time with the result x * 3
reactor.callLater(2, d.callback, x * 3)
return d
def printData(d):
"""
Data handling function to be added as a callback: handles the
data by printing the result
"""
print d
d = getDummyData(3)
d.addCallback(printData)
# manually set up the end of the process by asking the reactor to
# stop itself in 4 seconds time
reactor.callLater(4, reactor.stop)
# start up the Twisted reactor (event loop handler) manually
reactor.run()
Multiple callbacks
Multiple callbacks can be added to a Deferred. The first callback in theDeferred's callback chain will be called with the result, the second with theresult of the first callback, and so on. Why do we need this? Well, considera Deferred returned by twisted.enterprise.adbapi - the result of a SQL query.A web widget might add a callback that converts this result into HTML, andpass the Deferred onwards, where the callback will be used by twisted toreturn the result to the HTTP client. The callback chain will be bypassed incase of errors or exceptions.
from twisted.internet import reactor, defer
class Getter:
def gotResults(self, x):
"""
The Deferred mechanism provides a mechanism to signal error
conditions. In this case, odd numbers are bad.
This function demonstrates a more complex way of starting
the callback chain by checking for expected results and
choosing whether to fire the callback or errback chain
"""
if x % 2 == 0:
self.d.callback(x*3)
else:
self.d.errback(ValueError("You used an odd number!"))
def _toHTML(self, r):
"""
This function converts r to HTML.
It is added to the callback chain by getDummyData in
order to demonstrate how a callback passes its own result
to the next callback
"""
return "Result: %s" % r
def getDummyData(self, x):
"""
The Deferred mechanism allows for chained callbacks.
In this example, the output of gotResults is first
passed through _toHTML on its way to printData.
Again this function is a dummy, simulating a delayed result
using callLater, rather than using a real asynchronous
setup.
"""
self.d = defer.Deferred()
# simulate a delayed result by asking the reactor to schedule
# gotResults in 2 seconds time
reactor.callLater(2, self.gotResults, x)
self.d.addCallback(self._toHTML)
return self.d
def printData(d):
print d
def printError(failure):
import sys
sys.stderr.write(str(failure))
# this series of callbacks and errbacks will print an error message
g = Getter()
d = g.getDummyData(3)
d.addCallback(printData)
d.addErrback(printError)
# this series of callbacks and errbacks will print "Result: 12"
g = Getter()
d = g.getDummyData(4)
d.addCallback(printData)
d.addErrback(printError)
reactor.callLater(4, reactor.stop); reactor.run()
Visual Explanation
- Requesting method (data sink) requests data, gets Deferred object.
- Requesting method attaches callbacks to Deferred object.
- When the result is ready, give it to the Deferred object.
.callback(result)
if the operation succeeded,.errback(failure)
if it failed. Note thatfailure
is typically an instance of atwisted.python.failure.Failure
instance. - Deferred object triggers previously-added (call/err)back with the
result
orfailure
. Execution then follows the following rules, going down the chain of callbacks to be processed.- Result of the callback is always passed as the first argument to the next callback, creating a chain of processors.
- If a callback raises an exception, switch to errback.
- An unhandled failure gets passed down the line of errbacks, this creating an asynchronous analog to a series to a series of
except:
statements. - If an errback doesn't raise an exception or return a
twisted.python.failure.Failure
instance, switch to callback.
Errbacks
Deferred's error handling is modeled after Python'sexception handling. In the case that no errors occur, all thecallbacks run, one after the other, as described above.
If the errback is called instead of the callback (e.g. because a DB queryraised an error), then a twisted.python.failure.Failure
is passed into the firsterrback (you can add multiple errbacks, just like with callbacks). You canthink of your errbacks as being like except
blocksof ordinary Python code.
Unless you explicitly raise
an error in exceptblock, the Exception
is caught and stopspropagating, and normal execution continues. The same thing happens witherrbacks: unless you explicitly return
a Failure
or (re-)raise an exception, the error stopspropagating, and normal callbacks continue executing from that point (using thevalue returned from the errback). If the errback does returns a Failure
or raise an exception, then that is passed to thenext errback, and so on.
Note: If an errback doesn't return anything, then it effectivelyreturns None
, meaning that callbacks will continueto be executed after this errback. This may not be what you expect to happen,so be careful. Make sure your errbacks return a Failure
(probably the one that was passed to it), or ameaningful return value for the next callback.
Also, twisted.python.failure.Failure
instances havea useful method called trap, allowing you to effectively do the equivalentof:
try:
# code that may throw an exception
cookSpamAndEggs()
except (SpamException, EggException):
# Handle SpamExceptions and EggExceptions
...
You do this by:
def errorHandler(failure):
failure.trap(SpamException, EggException)
# Handle SpamExceptions and EggExceptions
d.addCallback(cookSpamAndEggs)
d.addErrback(errorHandler)
If none of arguments passed to failure.trap
match the error encapsulated in that Failure
, thenit re-raises the error.
There's another potential gotcha
here. There's amethod twisted.internet.defer.Deferred.addCallbacks
which is similar to, but not exactly the same as, addCallback
followed by addErrback
. In particular, consider these two cases:
# Case 1
d = getDeferredFromSomewhere()
d.addCallback(callback1) # A
d.addErrback(errback1) # B
d.addCallback(callback2)
d.addErrback(errback2)
# Case 2
d = getDeferredFromSomewhere()
d.addCallbacks(callback1, errback1) # C
d.addCallbacks(callback2, errback2)
If an error occurs in callback1
, then for Case 1errback1
will be called with the failure. For Case2, errback2
will be called. Be careful with yourcallbacks and errbacks.
What this means in a practical sense is in Case 1, "A" willhandle a success condition from getDeferredFromSomewhere
, and"B" will handle any errors that occur from either the upstreamsource, or that occur in 'A'. In Case 2, "C"'s errback1will only handle an error condition raised bygetDeferredFromSomewhere
, it will not do any handling oferrors raised in callback1.
Unhandled Errors
If a Deferred is garbage-collected with an unhandled error (i.e. it wouldcall the next errback if there was one), then Twisted will write the error'straceback to the log file. This means that you can typically get away with notadding errbacks and still get errors logged. Be careful though; if you keep areference to the Deferred around, preventing it from being garbage-collected,then you may never see the error (and your callbacks will mysteriously seem tohave never been called). If unsure, you should explicitly add an errback afteryour callbacks, even if all you do is:
# Make sure errors get logged
from twisted.python import log
d.addErrback(log.err)
Handling either synchronous or asynchronous results
In some applications, there are functions that might be either asynchronous orsynchronous. For example, a user authentication function might be able tocheck in memory whether a user is authenticated, allowing the authenticationfunction to return an immediate result, or it may need to wait onnetwork data, in which case it should return a Deferred to be firedwhen that data arrives. However, a function that wants to check if a user isauthenticated will then need to accept both immediate results andDeferreds.
In this example, the library function authenticateUser
uses theapplication function isValidUser
to authenticate a user:
def authenticateUser(isValidUser, user):
if isValidUser(user):
print "User is authenticated"
else:
print "User is not authenticated"
However, it assumes that isValidUser
returns immediately,whereas isValidUser
may actually authenticate the userasynchronously and return a Deferred. It is possible to adapt thistrivial user authentication code to accept either asynchronous isValidUser
or anasynchronous isValidUser
, allowing the library to handleeither type of function. It is, however, also possible to adaptsynchronous functions to return Deferreds. This section describes bothalternatives: handling functions that might be synchronous orasynchronous in the library function (authenticateUser
)or in the application code.
Handling possible Deferreds in the library code
Here is an example of a synchronous user authentication function that might bepassed to authenticateUser
:
def synchronousIsValidUser(user):
'''
Return true if user is a valid user, false otherwise
'''
return user in ["Alice", "Angus", "Agnes"]
However, here's an asynchronousIsValidUser
function that returnsa Deferred:
from twisted.internet import reactor
def asynchronousIsValidUser(d, user):
d = Deferred()
reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"])
return d
Our original implementation of authenticateUser
expectedisValidUser
to be synchronous, but now we need to change it to handle bothsynchronous and asynchronous implementations of isValidUser
. For this, weuse maybeDeferred
tocall isValidUser
, ensuring that the result of isValidUser
is a Deferred,even if isValidUser
is a synchronous function:
from twisted.internet import defer
def printResult(result):
if result:
print "User is authenticated"
else:
print "User is not authenticated"
def authenticateUser(isValidUser, user):
d = defer.maybeDeferred(isValidUser, user)
d.addCallback(printResult)
Now isValidUser
could be either synchronousIsValidUser
orasynchronousIsValidUser
.
It is also possible to modify synchronousIsValidUser
to returna Deferred, see Generating Deferreds for moreinformation.
DeferredList
Sometimes you want to be notified after several different events have allhappened, rather than waiting for each one individually. For example, you maywant to wait for all the connections in a list to close. twisted.internet.defer.DeferredList
is the way to dothis.
To create a DeferredList from multiple Deferreds, you simply pass a list ofthe Deferreds you want it to wait for:
# Creates a DeferredList
dl = defer.DeferredList([deferred1, deferred2, deferred3])
You can now treat the DeferredList like an ordinary Deferred; you can calladdCallbacks
and so on. The DeferredList will call its callbackwhen all the deferreds have completed. The callback will be called with a listof the results of the Deferreds it contains, like so:
def printResult(result):
print result
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
deferred3 = defer.Deferred()
dl = defer.DeferredList([deferred1, deferred2, deferred3])
dl.addCallback(printResult)
deferred1.callback('one')
deferred2.errback('bang!')
deferred3.callback('three')
# At this point, dl will fire its callback, printing:
# [(1, 'one'), (0, 'bang!'), (1, 'three')]
# (note that defer.SUCCESS == 1, and defer.FAILURE == 0)
A standard DeferredList will never call errback.
If you want to apply callbacks to the individual Deferreds thatgo into the DeferredList, you should be careful about when those callbacksare added. The act of adding a Deferred to a DeferredList inserts a callbackinto that Deferred (when that callback is run, it checks to see if theDeferredList has been completed yet). The important thing to remember isthat it is this callback which records the value that goes into theresult list handed to the DeferredList's callback.
Therefore, if you add a callback to the Deferred after adding theDeferred to the DeferredList, the value returned by that callback will notbe given to the DeferredList's callback. To avoid confusion, we recommend notadding callbacks to a Deferred once it has been used in a DeferredList.
def printResult(result):
print result
def addTen(result):
return result + " ten"
# Deferred gets callback before DeferredList is created
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
deferred1.addCallback(addTen)
dl = defer.DeferredList([deferred1, deferred2])
dl.addCallback(printResult)
deferred1.callback("one") # fires addTen, checks DeferredList, stores "one ten"
deferred2.callback("two")
# At this point, dl will fire its callback, printing:
# [(1, 'one ten'), (1, 'two')]
# Deferred gets callback after DeferredList is created
deferred1 = defer.Deferred()
deferred2 = defer.Deferred()
dl = defer.DeferredList([deferred1, deferred2])
deferred1.addCallback(addTen) # will run *after* DeferredList gets its value
dl.addCallback(printResult)
deferred1.callback("one") # checks DeferredList, stores "one", fires addTen
deferred2.callback("two")
# At this point, dl will fire its callback, printing:
# [(1, 'one), (1, 'two')]
Other behaviours
DeferredList accepts three keyword arguments that modify its behaviour:fireOnOneCallback
, fireOnOneErrback
andconsumeErrors
. If fireOnOneCallback
is set, theDeferredList will immediately call its callback as soon as any of its Deferredscall their callback. Similarly, fireOnOneErrback
will call errbackas soon as any of the Deferreds call their errback. Note that DeferredList isstill one-shot, like ordinary Deferreds, so after a callback or errback has beencalled the DeferredList will do nothing further (it will just silently ignoreany other results from its Deferreds).
The fireOnOneErrback
option is particularly useful when youwant to wait for all the results if everything succeeds, but also want to knowimmediately if something fails.
The consumeErrors
argument will stop the DeferredList frompropagating any errors along the callback chains of any Deferreds it contains(usually creating a DeferredList has no effect on the results passed along thecallbacks and errbacks of their Deferreds). Stopping errors at the DeferredListwith this option will prevent Unhandled error in Deferred
warnings fromthe Deferreds it contains without needing to add extra errbacks
Class Overview
This is an overview API reference for Deferred from the point of using aDeferred returned by a function. It is not meant to be asubstitute for the docstrings in the Deferred class, but can provide guidelinesfor its use.
There is a parallel overview of functions used by the Deferred'screator in Generating Deferreds.
Basic Callback Functions
addCallbacks(self, callback[, errback, callbackArgs, callbackKeywords, errbackArgs, errbackKeywords])
This is the method you will use to interact with Deferred. It adds a pair of callbacks
parallel
to each other (see diagram above) in the list of callbacks made when the Deferred is called back to. The signature of a method added using addCallbacks should bemyMethod(result, *methodArgs, **methodKeywords)
. If your method is passed in the callback slot, for example, all arguments in the tuplecallbackArgs
will be passed as*methodArgs
to your method.There are various convenience methods that are derivative of addCallbacks. I will not cover them in detail here, but it is important to know about them in order to create concise code.
addCallback(callback, *callbackArgs, **callbackKeywords)
Adds your callback at the next point in the processing chain, while adding an errback that will re-raise its first argument, not affecting further processing in the error case.
Note that, while addCallbacks (plural) requires the arguments to be passed in a tuple, addCallback (singular) takes all its remaining arguments as things to be passed to the callback function. The reason is obvious: addCallbacks (plural) cannot tell whether the arguments are meant for the callback or the errback, so they must be specifically marked by putting them into a tuple. addCallback (singular) knows that everything is destined to go to the callback, so it can use Python's
*
and**
syntax to collect the remaining arguments.addErrback(errback, *errbackArgs, **errbackKeywords)
Adds your errback at the next point in the processing chain, while adding a callback that will return its first argument, not affecting further processing in the success case.
addBoth(callbackOrErrback, *callbackOrErrbackArgs, **callbackOrErrbackKeywords)
This method adds the same callback into both sides of the processing chain at both points. Keep in mind that the type of the first argument is indeterminate if you use this method! Use it for
finally:
style blocks.
Chaining Deferreds
If you need one Deferred to wait on another, all you need to do is return aDeferred from a method added to addCallbacks. Specifically, if you returnDeferred B from a method added to Deferred A using A.addCallbacks, Deferred A'sprocessing chain will stop until Deferred B's .callback() method is called; atthat point, the next callback in A will be passed the result of the lastcallback in Deferred B's processing chain at the time.
If this seems confusing, don't worry about it right now -- when you run intoa situation where you need this behavior, you will probably recognize itimmediately and realize why this happens. If you want to chain deferredsmanually, there is also a convenience method to help you.
chainDeferred(otherDeferred)
Add
otherDeferred
to the end of this Deferred's processing chain. When self.callback is called, the result of my processing chain up to this point will be passed tootherDeferred.callback
. Further additions to my callback chain do not affectotherDeferred
This is the same as
self.addCallbacks(otherDeferred.callback, otherDeferred.errback)
See also
- Generating Deferreds, an introduction towriting asynchronous functions that return Deferreds.
Footnotes
- Unless of course a later callback starts a fresh error —but as we've already noted, adding callbacks to a Deferred after its used in aDeferredList is confusing and usually avoided.
Index
Version: 8.1.0- Twisted的延时处理机制
- twisted13 twisted的认证机制
- 理解Twisted的Deferred机制(一)
- 体会Twisted的异步处理过程
- python twisted reactor 的定时处理
- linux内核的延时机制
- twisted异步机制-Deferred
- twisted异步机制--Deferred
- 系统的延时及定时机制
- 在python的twisted中reactor信号处理
- 游戏手柄(JoyStick)的延时处理
- Android处理延时加载的方法
- Jquery hover方法的效果延时处理
- Android处理延时加载的方法
- Handler延时处理消息的流程
- java中延时任务的处理
- 从Twisted谈起异步处理
- twisted學習筆記二--twisted的那三板斧
- 应用程序日志管理!
- 编码字符集与乱码问题根源之所在
- 北京不仅仅10号线不通手机
- Home
- 浪费4个多小时修正因使用透明PNG而带来的问题
- Twisted的延时处理机制
- ANSI
- xhtml相对于html的好处
- Filesystems (ext3, reiser, xfs, jfs) comparison on Debian
- JS 复选框 全选 反选
- 节省时间
- 关于飞机加油问题
- 我要飞得更高系列一
- 魔兽世界出错排查以及解决方案