Coverage for crateweb/consent/celery.py: 80%

1""" 

2crate_anon/crateweb/consent/celery.py 

3 

4=============================================================================== 

5 

6 Copyright (C) 2015, University of Cambridge, Department of Psychiatry. 

7 Created by Rudolf Cardinal (rnc1001@cam.ac.uk). 

8 

9 This file is part of CRATE. 

10 

11 CRATE is free software: you can redistribute it and/or modify 

12 it under the terms of the GNU General Public License as published by 

13 the Free Software Foundation, either version 3 of the License, or 

14 (at your option) any later version. 

15 

16 CRATE is distributed in the hope that it will be useful, 

17 but WITHOUT ANY WARRANTY; without even the implied warranty of 

18 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 

19 GNU General Public License for more details. 

20 

21 You should have received a copy of the GNU General Public License 

22 along with CRATE. If not, see <https://www.gnu.org/licenses/>. 

23 

24=============================================================================== 

25 

26**Set up the Celery app for CRATE's web service back end.** 

27 

28*To test:* 

29 

301. Launch the Celery worker: 

31 

32.. code-block:: none 

33 

34 # no directory change required for an installed package 

35 # in early testing: 

36 # $ cd [...]/crateweb 

37 # $ celery --app consent worker --loglevel=debug 

38 # but with a proper package: 

39 

40 $ celery worker --app crate_anon.crateweb.consent --loglevel debug 

41 

422. Call the function asynchronously 

43 

44.. code-block:: none 

45 

46 $ manage.py shell 

47 > from consent.celery import debug_task 

48 > debug_task.delay() 

49 

50 or simply: 

51 $ python 

52 > from crate_anon.crateweb.consent.celery import debug_task 

53 > debug_task.delay() 

54 

55 

56**Command-line testing under Windows (2016-05-12):** 

57 

581. Run ``crate_launch_celery`` (which runs the "celery worker" command). 

59 With that running... 

60 

612. Do this: 

62 

63.. code-block:: none 

64 

65 $ python 

66 > from crate_anon.crateweb.consent.tasks import add 

67 > add(3, 4) # call immediately 

68 7 

69 > result = add.delay(3, 5) # call via Celery 

70 

71 # AT THAT MOMENT, THE CELERY WORKER PROCESS SHOULD SAY: 

72 # [... INFO/MainProcess] Received task: celery.local.add[...] 

73 # [... INFO/MainProcess] Task celery.local.add[...] succeeded in ...s: 8 

74 

75 > result 

76 <AsyncResult: 48ad8faa-ce75-4a19-bb0e-77feb9567b06> 

77 > result.ready() 

78 > result.state 

79 

803. When it doesn't work... 

81 

82.. code-block:: none 

83 

84 $ sudo rabbitmqctl list_queues name messages consumers 

85 

86.. code-block:: none 

87 

88 > from crate_anon.crateweb.consent.celery import app 

89 > app.control.purge() 

90 # Should report the number of tasks in the queue, after which the rabbitmqctl 

91 # should show an empty queue. 

92 

93 # Using the lengthy "--include" argument to "celery worker", which I was using 

94 # (in launch_celery.py) before creating a proper pip-installed package, seems 

95 # to be unnecessary. 

96 

97 # With "--loglevel info", the task list looks like: 

98 [tasks] 

99 . crate_anon.crateweb.consent.celery.debug_task 

100 . crate_anon.crateweb.consent.tasks.add 

101 . crate_anon.crateweb.consent.tasks.email_rdbm_task 

102 . crate_anon.crateweb.consent.tasks.finalize_clinician_response 

103 . crate_anon.crateweb.consent.tasks.process_consent_change 

104 . crate_anon.crateweb.consent.tasks.process_contact_request 

105 . crate_anon.crateweb.consent.tasks.process_patient_response 

106 . crate_anon.crateweb.consent.tasks.resend_email 

107 . crate_anon.crateweb.consent.tasks.test_email_rdbm_task 

108 

109 With "--loglevel debug", it looks like this: 

110 [tasks] 

111 . celery.backend_cleanup 

112 . celery.chain 

113 . celery.chord 

114 . celery.chord_unlock 

115 . celery.chunks 

116 . celery.group 

117 . celery.local.add 

118 . celery.local.email_rdbm_task 

119 . celery.local.finalize_clinician_response 

120 . celery.local.process_consent_change 

121 . celery.local.process_contact_request 

122 . celery.local.process_patient_response 

123 . celery.local.resend_email 

124 . celery.local.test_email_rdbm_task 

125 . celery.map 

126 . celery.starmap 

127 . crate_anon.crateweb.consent.celery.debug_task 

128 . crate_anon.crateweb.consent.tasks.add 

129 . crate_anon.crateweb.consent.tasks.email_rdbm_task 

130 . crate_anon.crateweb.consent.tasks.finalize_clinician_response 

131 . crate_anon.crateweb.consent.tasks.process_consent_change 

132 . crate_anon.crateweb.consent.tasks.process_contact_request 

133 . crate_anon.crateweb.consent.tasks.process_patient_response 

134 . crate_anon.crateweb.consent.tasks.resend_email 

135 . crate_anon.crateweb.consent.tasks.test_email_rdbm_task 

136 

137... but that doesn't stop it working under Linux. 

138 

139Inspecting workers: 

140 

141- https://docs.celeryproject.org/en/latest/userguide/workers.html#inspecting-workers # noqa: E501 

142 

143Inspecting lots of things: 

144 

145.. code-block:: bash 

146 

147 pip install flower 

148 celery -A crate_anon.crateweb.consent flower 

149 # now browse to http://localhost:5555/ 

150 

151Celery bug under Windows? https://github.com/celery/celery/issues/897 

152 

153Upgrade from Celery 3.1.19 to 3.1.23: 

154OK! Now we're talking. 

155 

156- Flower can now connect to the Celery worker (which reports things in its 

157 log as Flower does so). 

158- Tasks now show up in Flower -> Tasks, marked "Received", with appropriate 

159 arguments, and with a worker assigned, but the "Started" column remains 

160 blank. 

161 

162But then it stopped working again. 

163 

164- https://github.com/mher/flower/issues/452 

165 ... indicates that ``celery inspect ping`` should be doing something, but I'm 

166 getting "No workers replied within time constraint." On Linux, we get: 

167 

168 .. code-block:: none 

169 

170 -> celery@wombat: OK 

171 pong 

172 

173Improvement if you run the Celery worker with "--concurrency=4" (up from its 

174default of 1). Flower talks to the nodes. The tasks appear in the worker's 

175task list. Both "celery status" and "celery inspect ping" work. But tasks 

176still go into the "Reserved" list. 

177 

178The ``-Ofair`` option didn't fix it: 

179 

180- https://stackoverflow.com/questions/24519559 

181 

182And then it screwed up again, and became unresponsive to the status things. 

183Argh! 

184 

185- Close all Celery things. 

186- Hard reset of RabbitMQ: 

187 

188.. code-block:: bash 

189 

190 rabbitmqctl stop_app 

191 rabbitmqctl reset 

192 rabbitmqctl start_app 

193 

194- Start Celery worker(s). 

195- ``celery -A crate_anon.crateweb.consent status`` 

196 ... now works 

197- But tasks are still going into the reserved queue. 

198 

199People have achieved this with Win10: 

200 

201- https://github.com/hotdogee/django-blast/Windows-Development-Environment-Setup # noqa: E501 

202 

203Aha! Using ``--pool=solo`` made it work! 

204 

205- https://stackoverflow.com/questions/20309954 

206 

207 

208**See also...** 

209 

210See also :mod:`crate_anon.crateweb.consent.tasks`, with more notes. 

211 

212""" 

213 

214import os 

215 

216from celery import Celery, current_task 

217 

218# set the default Django settings module for the 'celery' program. 

219os.environ.setdefault( 

220 "DJANGO_SETTINGS_MODULE", "crate_anon.crateweb.config.settings" 

221) 

222# RNC: note that the module path must be accessible from the PYTHONPATH, 

223 

224# from django.conf import settings 

225 

226app = Celery("consent") 

227 

228# Using a string here means the worker will not have to 

229# pickle the object when using Windows. 

230app.config_from_object("django.conf:settings") 

231 

232# app.autodiscover_tasks(lambda: settings.INSTALLED_APPS) 

233# ... looks for "tasks.py" in all apps 

234# ... REQUIRES that all apps have an __init__.py 

235# https://github.com/celery/celery/issues/2523 

236 

237app.autodiscover_tasks(["crate_anon.crateweb.consent"]) # simpler! 

238 

239 

240@app.task(bind=True) 

241def debug_task(self) -> None: 

242 """ 

243 Debugging task for Celery that shows some request information only. 

244 

245 Args: 

246 self: the :class:`celery.Task` 

247 """ 

248 print(f"Request: {self.request!r}") 

249 print(f"Backend: {current_task.backend}")