Coverage for cc_modules/cc_cache.py: 100%

1# noinspection HttpUrlsUsage 

2""" 

3camcops_server/cc_modules/cc_cache.py 

4 

5=============================================================================== 

6 

7 Copyright (C) 2012, University of Cambridge, Department of Psychiatry. 

8 Created by Rudolf Cardinal (rnc1001@cam.ac.uk). 

9 

10 This file is part of CamCOPS. 

11 

12 CamCOPS is free software: you can redistribute it and/or modify 

13 it under the terms of the GNU General Public License as published by 

14 the Free Software Foundation, either version 3 of the License, or 

15 (at your option) any later version. 

16 

17 CamCOPS is distributed in the hope that it will be useful, 

18 but WITHOUT ANY WARRANTY; without even the implied warranty of 

19 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 

20 GNU General Public License for more details. 

21 

22 You should have received a copy of the GNU General Public License 

23 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>. 

24 

25=============================================================================== 

26 

27**Cache functions.** 

28 

291. The basic cache objects. 

30 

312. FIX FOR DOGPILE.CACHE FOR DECORATED FUNCTIONS, 2017-07-28 (PLUS SOME OTHER 

32 IMPROVEMENTS). SEE 

33 https://bitbucket.org/zzzeek/dogpile.cache/issues/96/error-in-python-35-with-use-of-deprecated 

34 

35Crash using type-hinted functions under Python 3.5 with dogpile.cache==0.6.4: 

36 

37.. code-block:: none 

38 

39 Traceback (most recent call last): 

40 File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main 

41 "__main__", mod_spec) 

42 File "/usr/lib/python3.5/runpy.py", line 85, in _run_code 

43 exec(code, run_globals) 

44 File "/home/rudolf/Documents/code/camcops/server/camcops_server/cc_modules/cc_cache.py", line 64, in <module> 

45 unit_test_cache() 

46 File "/home/rudolf/Documents/code/camcops/server/camcops_server/cc_modules/cc_cache.py", line 50, in unit_test_cache 

47 def testfunc() -> str: 

48 File "/home/rudolf/dev/venvs/camcops/lib/python3.5/site-packages/dogpile/cache/region.py", line 1215, in decorator 

49 key_generator = function_key_generator(namespace, fn) 

50 File "/home/rudolf/dev/venvs/camcops/lib/python3.5/site-packages/dogpile/cache/util.py", line 31, in function_key_generator 

51 args = inspect.getargspec(fn) 

52 File "/usr/lib/python3.5/inspect.py", line 1045, in getargspec 

53 raise ValueError("Function has keyword-only arguments or annotations" 

54 ValueError: Function has keyword-only arguments or annotations, use getfullargspec() API which can support them 

55 

563. CACHING NOTES 

57 

58- We currently use 'dogpile.cache.memory' as the backend. 

59 This means that for single-process (single-thread or multithreaded) servers, 

60 the cache is unique, but that won't work for multi-process (e.g. Gunicorn) 

61 servers. 

62 

63- That means that in a multiprocess environment it's fine to continue to use a 

64 memory cache for file-based stuff (e.g. configs, XML strings), but not for 

65 database-based stuff (e.g. which ID numbers are valid). 

66 

67- Correct solutions WITH a cache for those database-based things include: 

68 

69 - ignoring Python caching and relying on the MySQL query cache -- but this is 

70 being removed because it's not all that great: 

71 

72 https://mysqlserverteam.com/mysql-8-0-retiring-support-for-the-query-cache/ 

73 

74 - using memcached (via dogpile.cache.pylibmc) 

75 

76 https://www.ubergizmo.com/how-to/install-memcached-windows/ 

77 

78 - using redis (via dogpile.cache.redis and 

79 https://pypi.python.org/pypi/redis/ ) 

80 

81 https://stackoverflow.com/questions/10558465/memcached-vs-redis 

82 https://redis.io/ 

83 https://web.archive.org/web/20120118030804/http://simonwillison.net/static/2010/redis-tutorial/ 

84 http://oldblog.antirez.com/post/take-advantage-of-redis-adding-it-to-your-stack.html 

85 https://redis.io/topics/security 

86 

87 redis unsupported under Windows: 

88 https://redis.io/download 

89 

90- The other obvious alternative: don't cache such stuff! This may all be 

91 premature optimization. 

92 

93 https://msol.io/blog/tech/youre-probably-wrong-about-caching/ 

94 

95 The actual price is of the order of 0.6-1 ms per query, for the queries 

96 "find me all the ID number definitions" and "fetch the server settings". 

97 

98- The answer is probably: 

99 

100 - continue to use dogpile.cache.memory for simple "fixed" stuff read from 

101 disk; 

102 - continue to use Pyramid's per-request caching mechanism (@reify); 

103 - forget about database caching for now; 

104 - if it becomes a problem later, move to Redis 

105 

106- Therefore: 

107 

108 - there should be no calls to cache_region_static.delete 

109 

110""" # noqa 

111 

112 

113# ============================================================================= 

114# Imports; logging 

115# ============================================================================= 

116 

117from cardinal_pythonlib.dogpile_cache import kw_fkg_allowing_type_hints as fkg 

118from dogpile.cache import make_region 

119 

120 

121# ============================================================================= 

122# The main cache: static for the lifetime of this process. 

123# ============================================================================= 

124 

125cache_region_static = make_region() 

126cache_region_static.configure( 

127 backend="dogpile.cache.memory" 

128 # Consider later: memcached via dogpile.cache.pylibmc 

129) 

130 

131# Can now use: 

132# @cache_region_static.cache_on_arguments(function_key_generator=fkg) 

133 

134# https://stackoverflow.com/questions/44834/can-someone-explain-all-in-python 

135__all__ = ["cache_region_static", "fkg"] # prevents "Unused import statement"