Coverage for cc_modules/client_api.py: 91%
913 statements
« prev ^ index » next coverage.py v7.9.2, created at 2025-07-15 15:55 +0100
1"""
2camcops_server/cc_modules/client_api.py
4===============================================================================
6 Copyright (C) 2012, University of Cambridge, Department of Psychiatry.
7 Created by Rudolf Cardinal (rnc1001@cam.ac.uk).
9 This file is part of CamCOPS.
11 CamCOPS is free software: you can redistribute it and/or modify
12 it under the terms of the GNU General Public License as published by
13 the Free Software Foundation, either version 3 of the License, or
14 (at your option) any later version.
16 CamCOPS is distributed in the hope that it will be useful,
17 but WITHOUT ANY WARRANTY; without even the implied warranty of
18 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
19 GNU General Public License for more details.
21 You should have received a copy of the GNU General Public License
22 along with CamCOPS. If not, see <https://www.gnu.org/licenses/>.
24===============================================================================
26**Implements the API through which client devices (tablets etc.) upload and
27download data.**
29We use primarily SQLAlchemy Core here (in contrast to the ORM used elsewhere).
31This code is optimized to a degree for speed over clarity, aiming primarily to
32reduce the number of database hits.
34**The overall upload method is as follows**
36Everything that follows refers to records relating to a specific client device
37in the "current" era, only.
39In the preamble, the client:
41- verifies authorization via :func:`op_check_device_registered` and
42 :func:`op_check_upload_user_and_device`;
43- fetches and checks server ID information via :func:`op_get_id_info`;
44- checks its patients are acceptable via :func:`op_validate_patients`;
45- checks which tables are permitted via :func:`op_get_allowed_tables`;
46- performs some internal validity checks.
48Then, in the usual stepwise upload:
50- :func:`op_start_upload`
52 - Rolls back any previous incomplete changes via :func:`rollback_all`.
53 - Creates an upload batch, via :func:`get_batch_details_start_if_needed`.
55- If were are in a preserving/finalizing upload: :func:`op_start_preservation`.
57 - Marks all tables as dirty.
58 - Marks the upload batch as a "preserving" batch.
60- Then call some or all of:
62 - For tables that are empty on the client, :func:`op_upload_empty_tables`.
64 - Current client records are marked as ``_removal_pending``.
65 - Any table that had previous client records is marked as dirty.
66 - If preserving, any table without current records is marked as clean.
68 - For tables that the client wishes to send in one go,
69 :func:`op_upload_table`.
71 - Find current server records.
72 - Use :func:`upload_record_core` to add new records and modify existing
73 ones, and :func:`flag_deleted` to delete ones that weren't on the client.
74 - If any records are new, modified, or deleted, mark the table as dirty.
75 - If preserving and there were no server records in this table, mark the
76 table as clean.
78 - For tables (e.g. BLOBs) that might be too big to send in one go:
80 - client sends PKs to :func:`op_delete_where_key_not`, which "deletes" all
81 other records, via :func:`flag_deleted_where_clientpk_not`.
82 - client sends PK and timestamp values to :func:`op_which_keys_to_send`
83 - server "deletes" records that are not in the list (via
84 :func:`flag_deleted_where_clientpk_not`, which marks the table as dirty
85 if any records were thus modified). Note REDUNDANCY here re
86 :func:`op_delete_where_key_not`.
87 - server tells the client which records are new or need to be updated
88 - client sends each of those via :func:`op_upload_record`
90 - Calls :func`upload_record_core`.
91 - Marks the table as dirty, unless the client erroneously sent an
92 unchanged record.
94- In addition, specific records can be marked as ``_move_off_tablet``.
96 - :func:`upload_record_core` checks this for otherwise "identical" records
97 and applies that flag to the server.
99- When the client's finished, it calls :func:`op_end_upload`.
101 - Calls :func:`commit_all`;
102 - ... which, for all dirty tables, calls :func:`commit_table`;
103 - ... which executes the "add", "remove", and "preserve" functions for the
104 table;
105 - ... and triggers the updating of special server indexes on patient ID
106 numbers and tasks, via :func:`update_indexes`.
107 - At the end, :func:`commit_all` clears the dirty-table list.
109There's a little bit of special code to handle old tablet software, too.
111As of v2.3.0, the function :func:`op_upload_entire_database` does this in one
112step (faster; for use if the network packets are not excessively large).
114- Code relating to this uses ``batchdetails.onestep``.
116**Setup for the upload code**
118- Fire up a CamCOPS client with an empty database, e.g. from the build
119 directory via
121 .. code-block:: bash
123 ./camcops --dbdir ~/tmp/camcops_client_test
125- Fire up a web browser showing both (a) the task list via the index, and (b)
126 the task list without using the index. We'll use this to verify correct
127 indexing. **The two versions of the view should never be different.**
129- Ensure the test client device has no current records (force-finalize if
130 required).
132- Ensure the server's index is proper. Run ``camcops_server reindex`` if
133 required.
135- If required, fire up MySQL with the server database. You may wish to use
136 ``pager less -SFX``, for better display of large tables.
138**Testing the upload code**
140Perform the following steps both (1) with the client forced to the stepwise
141upload method, and (2) with it forced to one-step upload.
143Note that the number of patient ID numbers uploaded (etc.) is ignored below.
145*Single record*
147[Checked for one-step and multi-step upload, 2018-11-21.]
149#. Create a blank ReferrerSatisfactionSurvey (table ``ref_satis_gen``).
150 This has the advantage of being an anonymous single-record task.
152#. Upload/copy.
154 - The server log should show 1 × ref_satis_gen added.
156 - The task lists should show the task as current and incomplete.
158#. Modify it, so it's complete.
160#. Upload/copy.
162 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen
163 modified out.
165 - The task lists should show the task as current and complete.
167#. Upload/move.
169 - The server log should show 2 × ref_satis_gen preserved.
171 - The task lists should show the task as no longer current.
173#. Create another blank one.
175#. Upload/copy.
177#. Modify it so it's complete.
179#. Specifically flag it for preservation (the chequered flags).
181#. Upload/copy.
183 - The server log should show 1 × ref_satis_gen added, 1 × ref_satis_gen
184 modified out, 2 × ref_satis_gen preserved.
186 - The task lists should show the task as complete and no longer current.
188*With a patient*
190[Checked for one-step and multi-step upload, 2018-11-21.]
192#. Create a dummy patient that the server will accept.
194#. Create a Progress Note with location "loc1" and abort its creation, giving
195 an incomplete task.
197#. Create a second Progress Note with location "loc2" and contents "note2".
199#. Create a third Progress Note with location "loc3" and contents "note3".
201#. Upload/copy. Verify. This checks *addition*.
203 - The server log should show 1 × patient added; 3 × progressnote added.
204 (Also however many patientidnum records you chose.)
205 - All three tasks should be "current".
206 - The first should be "incomplete".
208#. Modify the first note by adding contents "note1".
210#. Delete the second note.
212#. Upload/copy. Verify. This checks *modification*, *deletion*,
213 *no-change detection*, and *reindexing*.
215 - The server log should show 1 × progressnote added, 1 × progressnote
216 modified out, 1 × progressnote deleted.
217 - The first note should now appear as complete.
218 - The second should have vanished.
219 - The third should be unchanged.
220 - The two remaining tasks should still be "current".
222#. Delete the contents from the first note again.
224#. Upload/move (or move-keeping-patients; that's only different on the
225 client side). Verify. This checks *preservation (finalizing)* and
226 *reindexing*.
228 - The server log should show 1 × patient preserved; 1 × progressnote added,
229 1 × progressnote modified out, 5 × progressnote preserved.
230 - The two remaining tasks should no longer be "current".
231 - The first should no longer be "complete".
233#. Create a complete "note 4" and an incomplete "note 5".
235#. Upload/copy.
237#. Force-finalize from the server. This tests force-finalizing including
238 reindexing.
240 - The "tasks to finalize" list should have just two tasks in it.
241 - After force-finalizing, the tasks should remain in the index but no
242 longer be marked as current.
244#. Upload/move to get rid of the residual tasks on the client.
246 - The server log should show 1 × patient added, 1 × patient preserved; 2 ×
247 progressnote added, 2 × progressnote preserved.
249*With ancillary tables and BLOBs*
251[Checked for one-step and multi-step upload, 2018-11-21.]
253#. Create a PhotoSequence with text "t1", one photo named "p1" of you holding
254 up one finger vertically, and another photo named "p2" of you holding up
255 two fingers vertically.
257#. Upload/copy.
259 - The server log should show:
261 - blobs: 2 × added;
262 - patient: 1 × added;
263 - photosequence: 1 × added;
264 - photosequence_photos: 2 × added.
266 - The task lists should look sensible.
268#. Clear the second photo and replace it with a photo of you holding up
269 two fingers horizontally.
271#. Upload/copy.
273 - The server log should show:
275 - blobs: 1 × added, 1 × modified out;
276 - photosequence: 1 × added, 1 × modified out;
277 - photosequence_photos: 1 × added, 1 × modified out.
279 - The task lists should look sensible.
281#. Back to two fingers vertically. (This is the fourth photo overall.)
283#. Mark that patient for specific finalization.
285#. Upload/copy.
287 - The server log should show:
289 - blobs: 1 × added, 1 × modified out, 4 × preserved;
290 - patient: 1 × preserved;
291 - photosequence: 1 × added, 1 × modified out, 3 × preserved;
292 - photosequence_photos: 1 × added, 1 × modified out, 4 × preserved.
294 - The tasks should no longer be current.
295 - A fresh "vertical fingers" photo should be visible.
297#. Create another patient and another PhotoSequence with one photo of three
298 fingers.
300#. Upload-copy.
302#. Force-finalize.
304 - Should finalize: 1 × blobs, 1 × patient, 1 × photosequence, 1 ×
305 photosequence_photos.
307#. Upload/move.
309During any MySQL debugging, remember:
311.. code-block:: none
313 -- For better display:
314 pager less -SFX;
316 -- To view relevant parts of the BLOB table without the actual BLOB:
318 SELECT
319 _pk, _group_id, _device_id, _era,
320 _current, _predecessor_pk, _successor_pk,
321 _addition_pending, _when_added_batch_utc, _adding_user_id,
322 _removal_pending, _when_removed_batch_utc, _removing_user_id,
323 _move_off_tablet,
324 _preserving_user_id, _forcibly_preserved,
325 id, tablename, tablepk, fieldname, mimetype, when_last_modified
326 FROM blobs;
328"""
330# =============================================================================
331# Imports
332# =============================================================================
334import logging
335import json
337# from pprint import pformat
338import secrets
339import string
340import time
341from typing import (
342 Any,
343 Dict,
344 Iterable,
345 List,
346 Optional,
347 Sequence,
348 Set,
349 Tuple,
350 TYPE_CHECKING,
351)
352from cardinal_pythonlib.datetimefunc import (
353 coerce_to_pendulum,
354 coerce_to_pendulum_date,
355 format_datetime,
356)
357from cardinal_pythonlib.httpconst import HttpMethod
358from cardinal_pythonlib.logs import BraceStyleAdapter
359from cardinal_pythonlib.pyramid.responses import TextResponse
360from cardinal_pythonlib.sqlalchemy.core_query import (
361 exists_in_table,
362 fetch_all_first_values,
363)
364from cardinal_pythonlib.text import escape_newlines
365from pendulum.exceptions import ParserError
366from pyramid.httpexceptions import HTTPBadRequest
367from pyramid.view import view_config
368from pyramid.response import Response
369from pyramid.security import NO_PERMISSION_REQUIRED
370from semantic_version import Version
371from sqlalchemy.engine import CursorResult
372from sqlalchemy.exc import IntegrityError
373from sqlalchemy.orm import joinedload
374from sqlalchemy.sql.expression import exists, select, update
375from sqlalchemy.sql.schema import Table
377from camcops_server.cc_modules import cc_audit # avoids "audit" name clash
378from camcops_server.cc_modules.cc_all_models import CLIENT_TABLE_MAP
379from camcops_server.cc_modules.cc_blob import Blob
380from camcops_server.cc_modules.cc_cache import cache_region_static, fkg
381from camcops_server.cc_modules.cc_client_api_core import (
382 AllowedTablesFieldNames,
383 BatchDetails,
384 exception_description,
385 ExtraStringFieldNames,
386 fail_server_error,
387 fail_unsupported_operation,
388 fail_user_error,
389 get_server_live_records,
390 require_keys,
391 ServerErrorException,
392 ServerRecord,
393 TabletParam,
394 UploadRecordResult,
395 UploadTableChanges,
396 UserErrorException,
397 values_delete_later,
398 values_delete_now,
399 values_preserve_now,
400 WhichKeyToSendInfo,
401)
402from camcops_server.cc_modules.cc_client_api_helpers import (
403 upload_commit_order_sorter,
404)
405from camcops_server.cc_modules.cc_constants import (
406 CLIENT_DATE_FIELD,
407 DateFormat,
408 ERA_NOW,
409 FP_ID_NUM,
410 FP_ID_DESC,
411 FP_ID_SHORT_DESC,
412 MOVE_OFF_TABLET_FIELD,
413 NUMBER_OF_IDNUMS_DEFUNCT, # allowed; for old tablet versions
414 POSSIBLE_SEX_VALUES,
415 TABLET_ID_FIELD,
416)
417from camcops_server.cc_modules.cc_convert import (
418 decode_single_value,
419 decode_values,
420 encode_single_value,
421)
422from camcops_server.cc_modules.cc_db import (
423 FN_ADDING_USER_ID,
424 FN_ADDITION_PENDING,
425 FN_CAMCOPS_VERSION,
426 FN_CURRENT,
427 FN_DEVICE_ID,
428 FN_ERA,
429 FN_GROUP_ID,
430 FN_PK,
431 FN_PREDECESSOR_PK,
432 FN_REMOVAL_PENDING,
433 FN_REMOVING_USER_ID,
434 FN_SUCCESSOR_PK,
435 FN_WHEN_ADDED_BATCH_UTC,
436 FN_WHEN_ADDED_EXACT,
437 FN_WHEN_REMOVED_BATCH_UTC,
438 FN_WHEN_REMOVED_EXACT,
439 RESERVED_FIELDS,
440)
441from camcops_server.cc_modules.cc_device import Device
442from camcops_server.cc_modules.cc_dirtytables import DirtyTable
443from camcops_server.cc_modules.cc_group import Group
444from camcops_server.cc_modules.cc_ipuse import IpUse
445from camcops_server.cc_modules.cc_membership import UserGroupMembership
446from camcops_server.cc_modules.cc_patient import (
447 Patient,
448 is_candidate_patient_valid_for_group,
449 is_candidate_patient_valid_for_restricted_user,
450)
451from camcops_server.cc_modules.cc_patientidnum import (
452 fake_tablet_id_for_patientidnum,
453 PatientIdNum,
454)
455from camcops_server.cc_modules.cc_proquint import (
456 InvalidProquintException,
457 uuid_from_proquint,
458)
459from camcops_server.cc_modules.cc_pyramid import Routes
460from camcops_server.cc_modules.cc_simpleobjects import (
461 BarePatientInfo,
462 IdNumReference,
463)
464from camcops_server.cc_modules.cc_specialnote import SpecialNote
465from camcops_server.cc_modules.cc_task import (
466 all_task_tables_with_min_client_version,
467)
468from camcops_server.cc_modules.cc_taskindex import (
469 update_indexes_and_push_exports,
470)
471from camcops_server.cc_modules.cc_user import User
472from camcops_server.cc_modules.cc_validators import (
473 STRING_VALIDATOR_TYPE,
474 validate_anything,
475 validate_email,
476)
477from camcops_server.cc_modules.cc_version import (
478 CAMCOPS_SERVER_VERSION_STRING,
479 MINIMUM_TABLET_VERSION,
480)
482if TYPE_CHECKING:
483 from camcops_server.cc_modules.cc_request import CamcopsRequest
485log = BraceStyleAdapter(logging.getLogger(__name__))
488# =============================================================================
489# Constants
490# =============================================================================
492DUPLICATE_FAILED = "Failed to duplicate record"
493INSERT_FAILED = "Failed to insert record"
495SUCCESS_MSG = "Success"
496SUCCESS_CODE = "1"
497FAILURE_CODE = "0"
499DEBUG_UPLOAD = False
502# =============================================================================
503# Quasi-constants
504# =============================================================================
506DB_JSON_DECODER = json.JSONDecoder() # just a plain one
507PATIENT_INFO_JSON_DECODER = json.JSONDecoder() # just a plain one
510# =============================================================================
511# Cached information
512# =============================================================================
515@cache_region_static.cache_on_arguments(function_key_generator=fkg)
516def all_tables_with_min_client_version() -> Dict[str, Version]:
517 """
518 For all tables that the client might upload to, return a mapping from the
519 table name to the corresponding minimum client version.
520 """
521 d = all_task_tables_with_min_client_version()
522 d[Blob.__tablename__] = MINIMUM_TABLET_VERSION
523 d[Patient.__tablename__] = MINIMUM_TABLET_VERSION
524 d[PatientIdNum.__tablename__] = MINIMUM_TABLET_VERSION
525 return d
528# =============================================================================
529# Validators
530# =============================================================================
533def ensure_valid_table_name(req: "CamcopsRequest", tablename: str) -> None:
534 """
535 Ensures a table name:
537 - doesn't contain bad characters,
538 - isn't a reserved table that the user is prohibited from accessing, and
539 - is a valid table name that's in the database.
541 Raises :exc:`UserErrorException` upon failure.
543 - 2017-10-08: shortcut to all that: it's OK if it's listed as a valid
544 client table.
545 - 2018-01-16 (v2.2.0): check also that client version is OK
546 """
547 if tablename not in CLIENT_TABLE_MAP:
548 fail_user_error(f"Invalid client table name: {tablename}")
549 tables_versions = all_tables_with_min_client_version()
550 assert tablename in tables_versions
551 client_version = req.tabletsession.tablet_version_ver
552 minimum_client_version = tables_versions[tablename]
553 if client_version < minimum_client_version:
554 fail_user_error(
555 f"Client CamCOPS version {client_version} is less than the "
556 f"version ({minimum_client_version}) "
557 f"required to handle table {tablename}"
558 )
561def ensure_valid_field_name(table: Table, fieldname: str) -> None:
562 """
563 Ensures a field name contains only valid characters, and isn't a
564 reserved fieldname that the user isn't allowed to access.
566 Raises :exc:`UserErrorException` upon failure.
568 - 2017-10-08: shortcut: it's OK if it's a column name for a particular
569 table.
570 """
571 if fieldname.startswith("_"): # all reserved fields start with _
572 # ... but not all fields starting with "_" are reserved; e.g.
573 # "_move_off_tablet" is allowed.
574 if fieldname in RESERVED_FIELDS:
575 fail_user_error(
576 f"Reserved field name for table {table.name!r}: {fieldname!r}"
577 )
578 if fieldname not in table.columns.keys():
579 fail_user_error(
580 f"Invalid field name for table {table.name!r}: {fieldname!r}"
581 )
582 # Note that the reserved-field check is case-sensitive, but so is the
583 # "present in table" check. So for a malicious uploader trying to use, for
584 # example, "_PK", this would not be picked up as a reserved field (so would
585 # pass that check) but then wouldn't be recognized as a valid field (so
586 # would fail).
589def ensure_string(value: Any, allow_none: bool = True) -> None:
590 """
591 Used when processing JSON information about patients: ensures that a value
592 is a string, or raises.
594 Args:
595 value: value to test
596 allow_none: is ``None`` allowed (not just an empty string)?
597 """
598 if value is None:
599 if allow_none:
600 return # OK
601 else:
602 fail_user_error("Patient JSON contains absent string")
603 if not isinstance(value, str):
604 fail_user_error(f"Patient JSON contains invalid non-string: {value!r}")
607def ensure_valid_patient_json(
608 req: "CamcopsRequest", group: Group, pt_dict: Dict[str, Any]
609) -> None:
610 """
611 Ensures that the JSON dictionary contains valid patient details (valid for
612 the group into which it's being uploaded), and that (if applicable) this
613 user is allowed to upload this patient.
615 Args:
616 req:
617 the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
618 group:
619 the :class:`camcops_server.cc_modules.cc_group.Group` into which
620 the upload is going
621 pt_dict:
622 a JSON dictionary from the client
624 Raises:
625 :exc:`UserErrorException` if invalid
627 """
628 if not isinstance(pt_dict, dict):
629 fail_user_error("Patient JSON is not a dict")
630 if not pt_dict:
631 fail_user_error("Patient JSON is empty")
632 valid_which_idnums = req.valid_which_idnums
633 errors = [] # type: List[str]
634 finalizing = None
635 ptinfo = BarePatientInfo()
636 idnum_types_seen = set() # type: Set[int]
637 for k, v in pt_dict.items():
638 # May not be necessary as JSON has already been validated
639 ensure_string(k, allow_none=False)
641 if k == TabletParam.FORENAME:
642 ensure_string(v)
643 ptinfo.forename = v
645 elif k == TabletParam.SURNAME:
646 ensure_string(v)
647 ptinfo.surname = v
649 elif k == TabletParam.SEX:
650 if v not in POSSIBLE_SEX_VALUES:
651 fail_user_error(f"Bad sex value: {v!r}")
652 ptinfo.sex = v
654 elif k == TabletParam.DOB:
655 ensure_string(v)
656 if v:
657 try:
658 # This will only return None if v is empty/None and we have
659 # already checked that
660 dob = coerce_to_pendulum_date(v)
661 except ParserError:
662 fail_user_error(f"Invalid DOB: {v!r}")
663 else:
664 dob = None
665 ptinfo.dob = dob
667 elif k == TabletParam.EMAIL:
668 ensure_string(v)
669 if v:
670 try:
671 validate_email(v)
672 except ValueError:
673 fail_user_error(f"Bad e-mail address: {v!r}")
674 ptinfo.email = v
676 elif k == TabletParam.ADDRESS:
677 ensure_string(v)
678 ptinfo.address = v
680 elif k == TabletParam.GP:
681 ensure_string(v)
682 ptinfo.gp = v
684 elif k == TabletParam.OTHER:
685 ensure_string(v)
686 ptinfo.otherdetails = v
688 elif k.startswith(TabletParam.IDNUM_PREFIX):
689 nstr = k[len(TabletParam.IDNUM_PREFIX) :]
690 try:
691 which_idnum = int(nstr)
692 except (TypeError, ValueError):
693 fail_user_error(f"Bad idnum key: {k!r}")
694 # noinspection PyUnboundLocalVariable
695 if which_idnum not in valid_which_idnums:
696 fail_user_error(f"Bad ID number type: {which_idnum}")
697 if which_idnum in idnum_types_seen:
698 fail_user_error(
699 f"More than one ID number supplied for ID "
700 f"number type {which_idnum}"
701 )
702 idnum_types_seen.add(which_idnum)
703 if v is not None and not isinstance(v, int):
704 fail_user_error(f"Bad ID number value: {v!r}")
705 idref = IdNumReference(which_idnum, v)
706 if not idref.is_valid():
707 fail_user_error(f"Bad ID number: {idref!r}")
708 ptinfo.add_idnum(idref)
710 elif k == TabletParam.FINALIZING:
711 if not isinstance(v, bool):
712 fail_user_error(f"Bad {k!r} value: {v!r}")
713 finalizing = v
715 else:
716 fail_user_error(f"Unknown JSON key: {k!r}")
718 if finalizing is None:
719 fail_user_error(f"Missing {TabletParam.FINALIZING!r} JSON key")
721 pt_ok, reason = is_candidate_patient_valid_for_group(
722 ptinfo, group, finalizing
723 )
724 if not pt_ok:
725 errors.append(f"{ptinfo} -> {reason}")
726 pt_ok, reason = is_candidate_patient_valid_for_restricted_user(req, ptinfo)
727 if not pt_ok:
728 errors.append(f"{ptinfo} -> {reason}")
729 if errors:
730 fail_user_error(f"Invalid patient: {' // '.join(errors)}")
733# =============================================================================
734# Extracting information from the POST request
735# =============================================================================
738def get_str_var(
739 req: "CamcopsRequest",
740 var: str,
741 mandatory: bool = True,
742 validator: STRING_VALIDATOR_TYPE = validate_anything,
743) -> Optional[str]:
744 """
745 Retrieves a string variable from the CamcopsRequest.
747 By default this performs no validation (because, for example, these strings
748 can contain SQL-encoded data or JSON), but there are a number of subsequent
749 operation-specific validation steps.
751 Args:
752 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
753 var: name of variable to retrieve
754 mandatory: if ``True``, raise an exception if the variable is missing
755 validator: validator function to use
757 Returns:
758 value
760 Raises:
761 :exc:`UserErrorException` if the variable was mandatory and
762 no value was provided
763 """
764 try:
765 val = req.get_str_param(var, default=None, validator=validator)
766 if mandatory and val is None:
767 fail_user_error(f"Must provide the variable: {var}")
768 return val
769 except HTTPBadRequest as e: # failed the validator
770 fail_user_error(str(e))
773def get_int_var(req: "CamcopsRequest", var: str) -> int:
774 """
775 Retrieves an integer variable from the CamcopsRequest.
777 Args:
778 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
779 var: name of variable to retrieve
781 Returns:
782 value
784 Raises:
785 :exc:`UserErrorException` if no value was provided, or if it wasn't an
786 integer
787 """
788 s = get_str_var(req, var, mandatory=True)
789 try:
790 return int(s)
791 except (TypeError, ValueError):
792 fail_user_error(f"Variable {var} is not a valid integer; was {s!r}")
795def get_bool_int_var(req: "CamcopsRequest", var: str) -> bool:
796 """
797 Retrieves a Boolean variable (encoded as an integer) from the
798 CamcopsRequest. Zero represents false; nonzero represents true.
800 Args:
801 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
802 var: name of variable to retrieve
804 Returns:
805 value
807 Raises:
808 :exc:`UserErrorException` if no value was provided, or if it wasn't an
809 integer
810 """
811 num = get_int_var(req, var)
812 return bool(num)
815def get_table_from_req(req: "CamcopsRequest", var: str) -> Table:
816 """
817 Retrieves a table name from a HTTP request, checks it's a valid client
818 table, and returns that table.
820 Args:
821 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
822 var: variable name (the variable's should be the table name)
824 Returns:
825 a SQLAlchemy :class:`Table`
827 Raises:
828 :exc:`UserErrorException` if the variable wasn't provided
829 """
830 tablename = get_str_var(req, var, mandatory=True)
831 ensure_valid_table_name(req, tablename)
832 return CLIENT_TABLE_MAP[tablename]
835def get_tables_from_post_var(
836 req: "CamcopsRequest", var: str, mandatory: bool = True
837) -> List[Table]:
838 """
839 Gets a list of tables from an HTTP request variable, and ensures all are
840 valid.
842 Args:
843 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
844 var: name of variable to retrieve
845 mandatory: if ``True``, raise an exception if the variable is missing
847 Returns:
848 a list of SQLAlchemy :class:`Table` objects
850 Raises:
851 :exc:`UserErrorException` if the variable was mandatory and
852 no value was provided, or if one or more tables was not valid
853 """
854 cstables = get_str_var(req, var, mandatory=mandatory)
855 if not cstables:
856 return []
857 # can't have any commas in table names, so it's OK to use a simple
858 # split() command
859 tablenames = [x.strip() for x in cstables.split(",")]
860 tables = [] # type: List[Table]
861 for tn in tablenames:
862 ensure_valid_table_name(req, tn)
863 tables.append(CLIENT_TABLE_MAP[tn])
864 return tables
867def get_single_field_from_post_var(
868 req: "CamcopsRequest", table: Table, var: str, mandatory: bool = True
869) -> str:
870 """
871 Retrieves a field (column) name from a the request and checks it's not a
872 bad fieldname for the specified table.
874 Args:
875 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
876 table: SQLAlchemy :class:`Table` in which the column should exist
877 var: name of variable to retrieve
878 mandatory: if ``True``, raise an exception if the variable is missing
880 Returns:
881 the field (column) name
883 Raises:
884 :exc:`UserErrorException` if the variable was mandatory and
885 no value was provided, or if the field was not valid for the specified
886 table
887 """
888 field = get_str_var(req, var, mandatory=mandatory)
889 ensure_valid_field_name(table, field)
890 return field
893def get_fields_from_post_var(
894 req: "CamcopsRequest",
895 table: Table,
896 var: str,
897 mandatory: bool = True,
898 allowed_nonexistent_fields: List[str] = None,
899) -> List[str]:
900 """
901 Get a comma-separated list of field names from a request and checks that
902 all are acceptable. Returns a list of fieldnames.
904 Args:
905 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
906 table: SQLAlchemy :class:`Table` in which the columns should exist
907 var: name of variable to retrieve
908 mandatory: if ``True``, raise an exception if the variable is missing
909 allowed_nonexistent_fields: fields that are allowed to be in the
910 upload but not in the database (special exemptions!)
912 Returns:
913 a list of the field (column) names
915 Raises:
916 :exc:`UserErrorException` if the variable was mandatory and
917 no value was provided, or if any field was not valid for the specified
918 table
919 """
920 csfields = get_str_var(req, var, mandatory=mandatory)
921 if not csfields:
922 return []
923 allowed_nonexistent_fields = (
924 allowed_nonexistent_fields or []
925 ) # type: List[str]
926 # can't have any commas in fields, so it's OK to use a simple
927 # split() command
928 fields = [x.strip() for x in csfields.split(",")]
929 for f in fields:
930 if f in allowed_nonexistent_fields:
931 continue
932 ensure_valid_field_name(table, f)
933 return fields
936def get_values_from_post_var(
937 req: "CamcopsRequest", var: str, mandatory: bool = True
938) -> List[Any]:
939 """
940 Retrieves a list of values from a CSV-separated list of SQL values
941 stored in a CGI form (including e.g. NULL, numbers, quoted strings, and
942 special handling for base-64/hex-encoded BLOBs.)
944 Args:
945 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
946 var: name of variable to retrieve
947 mandatory: if ``True``, raise an exception if the variable is missing
948 """
949 csvalues = get_str_var(req, var, mandatory=mandatory)
950 if not csvalues:
951 return []
952 return decode_values(csvalues)
955def get_fields_and_values(
956 req: "CamcopsRequest",
957 table: Table,
958 fields_var: str,
959 values_var: str,
960 mandatory: bool = True,
961) -> Dict[str, Any]:
962 """
963 Gets fieldnames and matching values from two variables in a request.
965 See :func:`get_fields_from_post_var`, :func:`get_values_from_post_var`.
967 Args:
968 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
969 table: SQLAlchemy :class:`Table` in which the columns should exist
970 fields_var: name of CSV "column names" variable to retrieve
971 values_var: name of CSV "corresponding values" variable to retrieve
972 mandatory: if ``True``, raise an exception if the variable is missing
974 Returns:
975 a dictionary mapping column names to decoded values
977 Raises:
978 :exc:`UserErrorException` if the variable was mandatory and
979 no value was provided, or if any field was not valid for the specified
980 table
981 """
982 fields = get_fields_from_post_var(
983 req, table, fields_var, mandatory=mandatory
984 )
985 values = get_values_from_post_var(req, values_var, mandatory=mandatory)
986 if len(fields) != len(values):
987 fail_user_error(
988 f"Number of fields ({len(fields)}) doesn't match number of values "
989 f"({len(values)})"
990 )
991 return dict(list(zip(fields, values)))
994def get_json_from_post_var(
995 req: "CamcopsRequest",
996 key: str,
997 decoder: json.JSONDecoder = None,
998 mandatory: bool = True,
999) -> Any:
1000 """
1001 Returns a Python object from a JSON-encoded value.
1003 Args:
1004 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1005 key: the name of the variable to retrieve
1006 decoder: the JSON decoder object to use; if ``None``, a default is
1007 created
1008 mandatory: if ``True``, raise an exception if the variable is missing
1010 Returns:
1011 Python object, e.g. a list of values, or ``None`` if the object is
1012 invalid and not mandatory
1014 Raises:
1015 :exc:`UserErrorException` if the variable was mandatory and
1016 no value was provided or the value was invalid JSON
1017 """
1018 decoder = decoder or json.JSONDecoder()
1019 j = get_str_var(req, key, mandatory=mandatory) # may raise
1020 if not j: # missing but not mandatory
1021 return None
1022 try:
1023 return decoder.decode(j)
1024 except json.JSONDecodeError:
1025 msg = f"Bad JSON for key {key!r}"
1026 if mandatory:
1027 fail_user_error(msg)
1028 else:
1029 log.warning(msg)
1030 return None
1033# =============================================================================
1034# Sending stuff to the client
1035# =============================================================================
1038def get_server_id_info(req: "CamcopsRequest") -> Dict[str, str]:
1039 """
1040 Returns a reply for the tablet, as a variable-to-value dictionary, giving
1041 details of the server.
1042 """
1043 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)
1044 reply = {
1045 TabletParam.DATABASE_TITLE: req.database_title,
1046 TabletParam.ID_POLICY_UPLOAD: group.upload_policy or "",
1047 TabletParam.ID_POLICY_FINALIZE: group.finalize_policy or "",
1048 TabletParam.SERVER_CAMCOPS_VERSION: CAMCOPS_SERVER_VERSION_STRING,
1049 }
1050 for iddef in req.idnum_definitions:
1051 n = iddef.which_idnum
1052 nstr = str(n)
1053 reply[TabletParam.ID_DESCRIPTION_PREFIX + nstr] = (
1054 iddef.description or ""
1055 )
1056 reply[TabletParam.ID_SHORT_DESCRIPTION_PREFIX + nstr] = (
1057 iddef.short_description or ""
1058 )
1059 reply[TabletParam.ID_VALIDATION_METHOD_PREFIX + nstr] = (
1060 iddef.validation_method or ""
1061 )
1062 return reply
1065def get_select_reply(
1066 fields: Sequence[str], rows: Sequence[Sequence[Any]]
1067) -> Dict[str, str]:
1068 """
1069 Formats the result of a ``SELECT`` query for the client as a dictionary
1070 reply.
1072 Args:
1073 fields: list of field names
1074 rows: list of rows, where each row is a list of values in the same
1075 order as ``fields``
1077 Returns:
1079 a dictionary of the format:
1081 .. code-block:: none
1083 {
1084 "nfields": NUMBER_OF_FIELDS,
1085 "fields": FIELDNAMES_AS_CSV,
1086 "nrecords": NUMBER_OF_RECORDS,
1087 "record0": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES,
1088 ...
1089 "record{nrecords - 1}": VALUES_AS_CSV_LIST_OF_ENCODED_SQL_VALUES
1090 }
1092 The final reply to the server is then formatted as text as per
1093 :func:`client_api`.
1095 """ # noqa
1096 nrecords = len(rows)
1097 reply = {
1098 TabletParam.NFIELDS: len(fields),
1099 TabletParam.FIELDS: ",".join(fields),
1100 TabletParam.NRECORDS: nrecords,
1101 }
1102 for r in range(nrecords):
1103 row = rows[r]
1104 encodedvalues = [] # type: List[str]
1105 for val in row:
1106 encodedvalues.append(encode_single_value(val))
1107 reply[TabletParam.RECORD_PREFIX + str(r)] = ",".join(encodedvalues)
1108 return reply # type: ignore[return-value]
1111# =============================================================================
1112# CamCOPS table reading functions
1113# =============================================================================
1116def record_exists(
1117 req: "CamcopsRequest",
1118 table: Table,
1119 clientpk_name: str,
1120 clientpk_value: Any,
1121) -> ServerRecord:
1122 """
1123 Checks if a record exists, using the device's perspective of a
1124 table/client PK combination.
1126 Args:
1127 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1128 table: an SQLAlchemy :class:`Table`
1129 clientpk_name: the column name of the client's PK
1130 clientpk_value: the client's PK value
1132 Returns:
1133 a :class:`ServerRecord` with the required information
1135 """
1136 query = (
1137 select(
1138 table.c[FN_PK], # server PK
1139 table.c[CLIENT_DATE_FIELD], # when last modified (on the server)
1140 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet
1141 )
1142 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1143 .where(table.c[FN_CURRENT])
1144 .where(table.c[FN_ERA] == ERA_NOW)
1145 .where(table.c[clientpk_name] == clientpk_value)
1146 )
1147 row = req.dbsession.execute(query).fetchone()
1148 if not row:
1149 return ServerRecord(clientpk_value, False)
1150 server_pk, server_when, move_off_tablet = row
1151 return ServerRecord(
1152 clientpk_value, True, server_pk, server_when, move_off_tablet
1153 )
1154 # Consider a warning/failure if we have >1 row meeting these criteria.
1155 # Not currently checked for.
1158def client_pks_that_exist(
1159 req: "CamcopsRequest",
1160 table: Table,
1161 clientpk_name: str,
1162 clientpk_values: List[int],
1163) -> Dict[int, ServerRecord]:
1164 """
1165 Searches for client PK values (for this device, current, and 'now')
1166 matching the input list.
1168 Args:
1169 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1170 table: an SQLAlchemy :class:`Table`
1171 clientpk_name: the column name of the client's PK
1172 clientpk_values: a list of the client's PK values
1174 Returns:
1175 a dictionary mapping client_pk to a :class:`ServerRecord` objects, for
1176 those records that match
1177 """
1178 query = (
1179 select(
1180 table.c[FN_PK], # server PK
1181 table.c[clientpk_name], # client PK
1182 table.c[CLIENT_DATE_FIELD], # when last modified (on the server)
1183 table.c[MOVE_OFF_TABLET_FIELD], # move_off_tablet
1184 )
1185 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1186 .where(table.c[FN_CURRENT])
1187 .where(table.c[FN_ERA] == ERA_NOW)
1188 .where(table.c[clientpk_name].in_(clientpk_values))
1189 )
1190 rows = req.dbsession.execute(query)
1191 d = {} # type: Dict[int, ServerRecord]
1192 for server_pk, client_pk, server_when, move_off_tablet in rows:
1193 d[client_pk] = ServerRecord(
1194 client_pk, True, server_pk, server_when, move_off_tablet
1195 )
1196 return d
1199def get_all_predecessor_pks(
1200 req: "CamcopsRequest",
1201 table: Table,
1202 last_pk: int,
1203 include_last: bool = True,
1204) -> List[int]:
1205 """
1206 Retrieves the PKs of all records that are predecessors of the specified one
1208 Args:
1209 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1210 table: an SQLAlchemy :class:`Table`
1211 last_pk: the PK to start with, and work backwards
1212 include_last: include ``last_pk`` in the list
1214 Returns:
1215 the PKs
1217 """
1218 dbsession = req.dbsession
1219 pks = [] # type: List[int]
1220 if include_last:
1221 pks.append(last_pk)
1222 current_pk = last_pk
1223 finished = False
1224 while not finished:
1225 next_pk = dbsession.execute(
1226 select(table.c[FN_PREDECESSOR_PK]).where(
1227 table.c[FN_PK] == current_pk
1228 )
1229 ).scalar() # type: Optional[int]
1230 if next_pk is None:
1231 finished = True
1232 else:
1233 pks.append(next_pk)
1234 current_pk = next_pk
1235 return sorted(pks)
1238# =============================================================================
1239# Record modification functions
1240# =============================================================================
1243def flag_deleted(
1244 req: "CamcopsRequest",
1245 batchdetails: BatchDetails,
1246 table: Table,
1247 pklist: Iterable[int],
1248) -> None:
1249 """
1250 Marks record(s) as deleted, specified by a list of server PKs within a
1251 table. (Note: "deleted" means "deleted with no successor", not "modified
1252 and replaced by a successor record".)
1253 """
1254 if batchdetails.onestep:
1255 values = values_delete_now(req, batchdetails)
1256 else:
1257 values = values_delete_later()
1258 req.dbsession.execute(
1259 update(table).where(table.c[FN_PK].in_(pklist)).values(values)
1260 )
1263def flag_all_records_deleted(req: "CamcopsRequest", table: Table) -> int:
1264 """
1265 Marks all records in a table as deleted (that are current and in the
1266 current era).
1268 Returns the number of rows affected.
1269 """
1270 rp = req.dbsession.execute(
1271 update(table)
1272 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1273 .where(table.c[FN_CURRENT])
1274 .where(table.c[FN_ERA] == ERA_NOW)
1275 .values(values_delete_later())
1276 ) # type: CursorResult
1277 return rp.rowcount
1278 # https://docs.sqlalchemy.org/en/latest/core/connections.html?highlight=rowcount#sqlalchemy.engine.Result.rowcount # noqa
1281def flag_deleted_where_clientpk_not(
1282 req: "CamcopsRequest",
1283 table: Table,
1284 clientpk_name: str,
1285 clientpk_values: Sequence[Any],
1286) -> None:
1287 """
1288 Marks for deletion all current/current-era records for a device, within a
1289 specific table, defined by a list of client-side PK values (and the name of
1290 the client-side PK column).
1291 """
1292 rp = req.dbsession.execute(
1293 update(table)
1294 .where(table.c[FN_DEVICE_ID] == req.tabletsession.device_id)
1295 .where(table.c[FN_CURRENT])
1296 .where(table.c[FN_ERA] == ERA_NOW)
1297 .where(table.c[clientpk_name].notin_(clientpk_values))
1298 .values(values_delete_later())
1299 ) # type: CursorResult
1300 if rp.rowcount > 0:
1301 mark_table_dirty(req, table)
1302 # ... but if we are preserving, do NOT mark this table as clean; there may
1303 # be other records that still require preserving.
1306def flag_modified(
1307 req: "CamcopsRequest",
1308 batchdetails: BatchDetails,
1309 table: Table,
1310 pk: int,
1311 successor_pk: int,
1312) -> None:
1313 """
1314 Marks a record as old, storing its successor's details.
1316 Args:
1317 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1318 batchdetails: the :class:`BatchDetails`
1319 table: SQLAlchemy :class:`Table`
1320 pk: server PK of the record to mark as old
1321 successor_pk: server PK of its successor
1322 """
1323 if batchdetails.onestep:
1324 req.dbsession.execute(
1325 update(table)
1326 .where(table.c[FN_PK] == pk)
1327 .values(
1328 {
1329 FN_CURRENT: 0,
1330 FN_REMOVAL_PENDING: 0,
1331 FN_SUCCESSOR_PK: successor_pk,
1332 FN_REMOVING_USER_ID: req.user_id,
1333 FN_WHEN_REMOVED_EXACT: req.now,
1334 FN_WHEN_REMOVED_BATCH_UTC: batchdetails.batchtime,
1335 }
1336 )
1337 )
1338 else:
1339 req.dbsession.execute(
1340 update(table)
1341 .where(table.c[FN_PK] == pk)
1342 .values({FN_REMOVAL_PENDING: 1, FN_SUCCESSOR_PK: successor_pk})
1343 )
1346def flag_multiple_records_for_preservation(
1347 req: "CamcopsRequest",
1348 batchdetails: BatchDetails,
1349 table: Table,
1350 pks_to_preserve: List[int],
1351) -> None:
1352 """
1353 Low-level function to mark records for preservation by server PK.
1354 Does not concern itself with the predecessor chain (for which, see
1355 :func:`flag_record_for_preservation`).
1357 Args:
1358 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1359 batchdetails: the :class:`BatchDetails`
1360 table: SQLAlchemy :class:`Table`
1361 pks_to_preserve: server PK of the records to mark as preserved
1362 """
1363 if batchdetails.onestep:
1364 req.dbsession.execute(
1365 update(table)
1366 .where(table.c[FN_PK].in_(pks_to_preserve))
1367 .values(values_preserve_now(req, batchdetails))
1368 )
1369 # Also any associated special notes:
1370 new_era = batchdetails.new_era
1371 # noinspection PyUnresolvedReferences
1372 req.dbsession.execute(
1373 update(SpecialNote.__table__) # type: ignore[arg-type]
1374 .where(SpecialNote.basetable == table.name)
1375 .where(SpecialNote.device_id == req.tabletsession.device_id)
1376 .where(SpecialNote.era == ERA_NOW)
1377 .where(
1378 exists()
1379 .select_from(table)
1380 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)
1381 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)
1382 .where(table.c[FN_ERA] == new_era)
1383 )
1384 # ^^^^^^^^^^^^^^^^^^^^^^^^^^
1385 # This bit restricts to records being preserved.
1386 .values(era=new_era)
1387 )
1388 else:
1389 req.dbsession.execute(
1390 update(table)
1391 .where(table.c[FN_PK].in_(pks_to_preserve))
1392 .values({MOVE_OFF_TABLET_FIELD: 1})
1393 )
1396def flag_record_for_preservation(
1397 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table, pk: int
1398) -> List[int]:
1399 """
1400 Marks a record for preservation (moving off the tablet, changing its
1401 era details).
1403 2018-11-18: works back through the predecessor chain too, fixing an old
1404 bug.
1406 Args:
1407 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1408 batchdetails: the :class:`BatchDetails`
1409 table: SQLAlchemy :class:`Table`
1410 pk: server PK of the record to mark
1412 Returns:
1413 list: all PKs being preserved
1414 """
1415 pks_to_preserve = get_all_predecessor_pks(req, table, pk)
1416 flag_multiple_records_for_preservation(
1417 req, batchdetails, table, pks_to_preserve
1418 )
1419 return pks_to_preserve
1422def preserve_all(
1423 req: "CamcopsRequest", batchdetails: BatchDetails, table: Table
1424) -> None:
1425 """
1426 Preserves all records in a table for a device, including non-current ones.
1428 Args:
1429 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1430 batchdetails: the :class:`BatchDetails`
1431 table: SQLAlchemy :class:`Table`
1432 """
1433 device_id = req.tabletsession.device_id
1434 req.dbsession.execute(
1435 update(table)
1436 .where(table.c[FN_DEVICE_ID] == device_id)
1437 .where(table.c[FN_ERA] == ERA_NOW)
1438 .values(values_preserve_now(req, batchdetails))
1439 )
1442# =============================================================================
1443# Upload helper functions
1444# =============================================================================
1447def process_upload_record_special(
1448 req: "CamcopsRequest",
1449 batchdetails: BatchDetails,
1450 table: Table,
1451 valuedict: Dict[str, Any],
1452) -> None:
1453 """
1454 Special processing function for upload, in which we inspect the data.
1455 Called by :func:`upload_record_core`.
1457 1. Handles old clients with ID information in the patient table, etc.
1458 (Note: this can be IGNORED for any client using
1459 :func:`op_upload_entire_database`, as these are newer.)
1461 2. Validates ID numbers.
1463 Args:
1464 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1465 batchdetails: the :class:`BatchDetails`
1466 table: an SQLAlchemy :class:`Table`
1467 valuedict: a dictionary of {colname: value} pairs from the client.
1468 May be modified.
1469 """
1470 ts = req.tabletsession
1471 tablename = table.name
1473 if tablename == Patient.__tablename__:
1474 # ---------------------------------------------------------------------
1475 # Deal with old tablets that had ID numbers in a less flexible format.
1476 # ---------------------------------------------------------------------
1477 if ts.cope_with_deleted_patient_descriptors:
1478 # Old tablets (pre-2.0.0) will upload copies of the ID
1479 # descriptions with the patient. To cope with that, we
1480 # remove those here:
1481 for n in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
1482 nstr = str(n)
1483 fn_desc = FP_ID_DESC + nstr
1484 fn_shortdesc = FP_ID_SHORT_DESC + nstr
1485 valuedict.pop(fn_desc, None) # remove item, if exists
1486 valuedict.pop(fn_shortdesc, None)
1488 if ts.cope_with_old_idnums:
1489 # Insert records into the new ID number table from the old
1490 # patient table:
1491 for which_idnum in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
1492 nstr = str(which_idnum)
1493 fn_idnum = FP_ID_NUM + nstr
1494 idnum_value = valuedict.pop(fn_idnum, None)
1495 # ... and remove it from our new Patient record
1496 patient_id = valuedict.get("id", None)
1497 if idnum_value is None or patient_id is None:
1498 continue
1499 # noinspection PyUnresolvedReferences
1500 mark_table_dirty(req, PatientIdNum.__table__) # type: ignore[arg-type] # noqa: E501
1501 client_date_value = coerce_to_pendulum(
1502 valuedict[CLIENT_DATE_FIELD]
1503 )
1504 # noinspection PyUnresolvedReferences
1505 upload_record_core(
1506 req=req,
1507 batchdetails=batchdetails,
1508 table=PatientIdNum.__table__, # type: ignore[arg-type]
1509 clientpk_name="id",
1510 valuedict={
1511 "id": fake_tablet_id_for_patientidnum(
1512 patient_id=patient_id, which_idnum=which_idnum
1513 ), # ... guarantees a pseudo client PK
1514 "patient_id": patient_id,
1515 "which_idnum": which_idnum,
1516 "idnum_value": idnum_value,
1517 CLIENT_DATE_FIELD: client_date_value,
1518 MOVE_OFF_TABLET_FIELD: valuedict[
1519 MOVE_OFF_TABLET_FIELD
1520 ],
1521 },
1522 )
1523 # Now, how to deal with deletion, i.e. records missing from the
1524 # tablet? See our caller, op_upload_table(), which has a special
1525 # handler for this.
1526 #
1527 # Note that op_upload_record() is/was only used for BLOBs, so we
1528 # don't have to worry about special processing for that aspect
1529 # here; also, that method handles deletion in a different way.
1531 elif tablename == PatientIdNum.__tablename__:
1532 # ---------------------------------------------------------------------
1533 # Validate ID numbers.
1534 # ---------------------------------------------------------------------
1535 which_idnum = valuedict.get("which_idnum", None)
1536 if which_idnum not in req.valid_which_idnums:
1537 fail_user_error(f"No such ID number type: {which_idnum}")
1538 idnum_value = valuedict.get("idnum_value", None)
1539 if not req.is_idnum_valid(which_idnum, idnum_value):
1540 why_invalid = req.why_idnum_invalid(which_idnum, idnum_value)
1541 fail_user_error(
1542 f"For ID type {which_idnum}, ID number {idnum_value} is "
1543 f"invalid: {why_invalid}"
1544 )
1547def upload_record_core(
1548 req: "CamcopsRequest",
1549 batchdetails: BatchDetails,
1550 table: Table,
1551 clientpk_name: str,
1552 valuedict: Dict[str, Any],
1553 server_live_current_records: List[ServerRecord] = None,
1554) -> UploadRecordResult:
1555 """
1556 Uploads a record. Deals with IDENTICAL, NEW, and MODIFIED records.
1558 Used by :func:`upload_table` and :func:`upload_record`.
1560 Args:
1561 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1562 batchdetails: the :class:`BatchDetails`
1563 table: an SQLAlchemy :class:`Table`
1564 clientpk_name: the column name of the client's PK
1565 valuedict: a dictionary of {colname: value} pairs from the client
1566 server_live_current_records: list of :class:`ServerRecord` objects for
1567 the active records on the server for this client, in this table
1569 Returns:
1570 a :class:`UploadRecordResult` object
1571 """
1572 require_keys(
1573 valuedict, [clientpk_name, CLIENT_DATE_FIELD, MOVE_OFF_TABLET_FIELD]
1574 )
1575 clientpk_value = valuedict[clientpk_name]
1577 if server_live_current_records:
1578 # All server records for this table/device/era have been prefetched.
1579 serverrec = next(
1580 (
1581 r
1582 for r in server_live_current_records
1583 if r.client_pk == clientpk_value
1584 ),
1585 None,
1586 )
1587 if serverrec is None:
1588 serverrec = ServerRecord(clientpk_value, False)
1589 else:
1590 # Look up this record specifically.
1591 serverrec = record_exists(req, table, clientpk_name, clientpk_value)
1593 if DEBUG_UPLOAD:
1594 log.debug("upload_record_core: {}, {}", table.name, serverrec)
1596 oldserverpk = serverrec.server_pk
1597 urr = UploadRecordResult(
1598 oldserverpk=oldserverpk,
1599 specifically_marked_for_preservation=bool(
1600 valuedict[MOVE_OFF_TABLET_FIELD]
1601 ),
1602 dirty=True,
1603 )
1604 if serverrec.exists:
1605 # There's an existing record, which is either identical or not.
1606 client_date_value = coerce_to_pendulum(valuedict[CLIENT_DATE_FIELD])
1607 if serverrec.server_when == client_date_value:
1608 # The existing record is identical.
1609 # No action needed unless MOVE_OFF_TABLET_FIELDNAME is set.
1610 if not urr.specifically_marked_for_preservation:
1611 urr.dirty = False
1612 else:
1613 # The existing record is different. We need a logical UPDATE, but
1614 # maintaining an audit trail.
1615 process_upload_record_special(req, batchdetails, table, valuedict)
1616 urr.newserverpk = insert_record(
1617 req, batchdetails, table, valuedict, oldserverpk
1618 )
1619 flag_modified(
1620 req, batchdetails, table, oldserverpk, urr.newserverpk
1621 )
1622 else:
1623 # The record is NEW. We need to INSERT it.
1624 process_upload_record_special(req, batchdetails, table, valuedict)
1625 urr.newserverpk = insert_record(
1626 req, batchdetails, table, valuedict, None
1627 )
1628 if urr.specifically_marked_for_preservation:
1629 preservation_pks = flag_record_for_preservation(
1630 req, batchdetails, table, urr.latest_pk
1631 )
1632 urr.note_specifically_marked_preservation_pks(preservation_pks)
1634 if DEBUG_UPLOAD:
1635 log.debug("upload_record_core: {}, {!r}", table.name, urr)
1636 return urr
1639def insert_record(
1640 req: "CamcopsRequest",
1641 batchdetails: BatchDetails,
1642 table: Table,
1643 valuedict: Dict[str, Any],
1644 predecessor_pk: Optional[int],
1645) -> int:
1646 """
1647 Inserts a record, or raises an exception if that fails.
1649 Args:
1650 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1651 batchdetails: the :class:`BatchDetails`
1652 table: an SQLAlchemy :class:`Table`
1653 valuedict: a dictionary of {colname: value} pairs from the client
1654 predecessor_pk: an optional server PK of the record's predecessor
1656 Returns:
1657 the server PK of the new record
1658 """
1659 ts = req.tabletsession
1660 valuedict.update(
1661 {
1662 FN_DEVICE_ID: ts.device_id,
1663 FN_ERA: ERA_NOW,
1664 FN_REMOVAL_PENDING: 0,
1665 FN_PREDECESSOR_PK: predecessor_pk,
1666 FN_CAMCOPS_VERSION: ts.tablet_version_str,
1667 FN_GROUP_ID: req.user.upload_group_id,
1668 }
1669 )
1670 if batchdetails.onestep:
1671 valuedict.update(
1672 {
1673 FN_CURRENT: 1,
1674 FN_ADDITION_PENDING: 0,
1675 FN_ADDING_USER_ID: req.user_id,
1676 FN_WHEN_ADDED_EXACT: req.now,
1677 FN_WHEN_ADDED_BATCH_UTC: batchdetails.batchtime,
1678 }
1679 )
1680 else:
1681 valuedict.update({FN_CURRENT: 0, FN_ADDITION_PENDING: 1})
1682 rp = req.dbsession.execute(
1683 table.insert().values(valuedict)
1684 ) # type: CursorResult
1685 # In SQLAlchemy 1.3, execute() returned a ResultProxy, and after an
1686 # insert() call, ResultProxy.inserted_primary_key was a list of scalars,
1687 # corresponding to the list of primary key columns in the target table,
1688 # representing the primary key of the row just inserted (a list because a
1689 # primary key can be a composite of many columns) [1]. We then asserted it
1690 # was a list of length 1, and returned the first element. In SQLAlchemy
1691 # 1.4+, we get a CursorResult back instead, and its inserted_primary_key is
1692 # a named tuple of primary key values, for that single inserted row [2] (or
1693 # None if there was not a valid single-row insert, or raises an exception
1694 # after a multi-row insert) [3]. The previous length check was likely an
1695 # inaccurate attempt to check that 1 row had been inserted (rather than
1696 # that there was 1 primary key column). If the insert fails, however, the
1697 # database call will raise an exception; and even if it didn't, the attempt
1698 # to access rp.inserted_primary_key[0] as None[0] would raise a TypeError.
1699 # So the additional assertion was a waste of time.
1700 # [1] https://docs.sqlalchemy.org/en/13/core/connections.html#sqlalchemy.engine.ResultProxy.inserted_primary_key # noqa: E501
1701 # [2] https://docs.sqlalchemy.org/en/14/core/connections.html#sqlalchemy.engine.BaseCursorResult.inserted_primary_key # noqa: E501
1702 # [3] see sqlalchemy/engine/cursor.py
1703 return rp.inserted_primary_key[0]
1706def audit_upload(
1707 req: "CamcopsRequest", changes: List[UploadTableChanges]
1708) -> None:
1709 """
1710 Writes audit information for an upload.
1712 Args:
1713 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1714 changes: a list of :class:`UploadTableChanges` objects, one per table
1715 """
1716 msg = (
1717 f"Upload from device {req.tabletsession.device_id}, "
1718 f"username {req.tabletsession.username!r}: "
1719 )
1720 changes = [x for x in changes if x.any_changes]
1721 if changes:
1722 changes.sort(key=lambda x: x.tablename)
1723 msg += ", ".join(x.description() for x in changes)
1724 else:
1725 msg += "No changes"
1726 log.info("audit_upload: {}", msg)
1727 audit(req, msg)
1730# =============================================================================
1731# Batch (atomic) upload and preserving
1732# =============================================================================
1735def get_batch_details(req: "CamcopsRequest") -> BatchDetails:
1736 """
1737 Returns the :class:`BatchDetails` for the current upload. If none exists,
1738 a new batch is created and returned.
1740 SIDE EFFECT: if the username is different from the username that started
1741 a previous upload batch for this device, we restart the upload batch (thus
1742 rolling back previous pending changes).
1744 Raises:
1745 :exc:`camcops_server.cc_modules.cc_client_api_core.ServerErrorException`
1746 if the device doesn't exist
1747 """
1748 device_id = req.tabletsession.device_id
1749 # noinspection PyUnresolvedReferences
1750 query = (
1751 select(
1752 Device.ongoing_upload_batch_utc,
1753 Device.uploading_user_id,
1754 Device.currently_preserving,
1755 )
1756 .select_from(Device.__table__)
1757 .where(Device.id == device_id)
1758 )
1759 row = req.dbsession.execute(query).fetchone()
1760 if not row:
1761 fail_server_error(
1762 f"Device {device_id} missing from Device table"
1763 ) # will raise
1764 upload_batch_utc, uploading_user_id, currently_preserving = row
1765 if not upload_batch_utc or uploading_user_id != req.user_id:
1766 # SIDE EFFECT: if the username changes, we restart (and thus roll back
1767 # previous pending changes)
1768 start_device_upload_batch(req)
1769 return BatchDetails(req.now_utc, False)
1770 return BatchDetails(upload_batch_utc, currently_preserving)
1773def start_device_upload_batch(req: "CamcopsRequest") -> None:
1774 """
1775 Starts an upload batch for a device.
1776 """
1777 rollback_all(req)
1778 # noinspection PyUnresolvedReferences
1779 req.dbsession.execute(
1780 update(Device.__table__) # type: ignore[arg-type]
1781 .where(Device.id == req.tabletsession.device_id)
1782 .values(
1783 last_upload_batch_utc=req.now_utc,
1784 ongoing_upload_batch_utc=req.now_utc,
1785 uploading_user_id=req.tabletsession.user_id,
1786 )
1787 )
1790def _clear_ongoing_upload_batch_details(req: "CamcopsRequest") -> None:
1791 """
1792 Clears upload batch details from the Device table.
1794 Args:
1795 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1796 """
1797 # noinspection PyUnresolvedReferences
1798 req.dbsession.execute(
1799 update(Device.__table__) # type: ignore[arg-type]
1800 .where(Device.id == req.tabletsession.device_id)
1801 .values(
1802 ongoing_upload_batch_utc=None,
1803 uploading_user_id=None,
1804 currently_preserving=0,
1805 )
1806 )
1809def end_device_upload_batch(
1810 req: "CamcopsRequest", batchdetails: BatchDetails
1811) -> None:
1812 """
1813 Ends an upload batch, committing all changes made thus far.
1815 Args:
1816 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1817 batchdetails: the :class:`BatchDetails`
1818 """
1819 commit_all(req, batchdetails)
1820 _clear_ongoing_upload_batch_details(req)
1823def clear_device_upload_batch(req: "CamcopsRequest") -> None:
1824 """
1825 Ensures there is nothing pending. Rools back previous changes. Wipes any
1826 ongoing batch details.
1828 Args:
1829 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1830 """
1831 rollback_all(req)
1832 _clear_ongoing_upload_batch_details(req)
1835def start_preserving(req: "CamcopsRequest") -> None:
1836 """
1837 Starts preservation (the process of moving records from the NOW era to
1838 an older era, so they can be removed safely from the tablet).
1840 Called by :func:`op_start_preservation`.
1842 In this situation, we start by assuming that ALL tables are "dirty",
1843 because they may have live records from a previous upload.
1844 """
1845 # noinspection PyUnresolvedReferences
1846 req.dbsession.execute(
1847 update(Device.__table__) # type: ignore[arg-type]
1848 .where(Device.id == req.tabletsession.device_id)
1849 .values(currently_preserving=1)
1850 )
1851 mark_all_tables_dirty(req)
1854def mark_table_dirty(req: "CamcopsRequest", table: Table) -> None:
1855 """
1856 Marks a table as having been modified during the current upload.
1857 """
1858 tablename = table.name
1859 device_id = req.tabletsession.device_id
1860 dbsession = req.dbsession
1861 # noinspection PyUnresolvedReferences
1862 table_already_dirty = exists_in_table(
1863 dbsession,
1864 DirtyTable.__table__, # type: ignore[arg-type]
1865 DirtyTable.device_id == device_id,
1866 DirtyTable.tablename == tablename,
1867 )
1868 if not table_already_dirty:
1869 # noinspection PyUnresolvedReferences
1870 dbsession.execute(
1871 DirtyTable.__table__.insert().values( # type: ignore[attr-defined]
1872 device_id=device_id, tablename=tablename
1873 )
1874 )
1877def mark_tables_dirty(req: "CamcopsRequest", tables: List[Table]) -> None:
1878 """
1879 Marks multiple tables as dirty.
1880 """
1881 if not tables:
1882 return
1883 device_id = req.tabletsession.device_id
1884 tablenames = [t.name for t in tables]
1885 # Delete first
1886 # noinspection PyUnresolvedReferences
1887 req.dbsession.execute(
1888 DirtyTable.__table__.delete() # type: ignore[attr-defined]
1889 .where(DirtyTable.device_id == device_id)
1890 .where(DirtyTable.tablename.in_(tablenames))
1891 )
1892 # Then insert
1893 insert_values = [
1894 {"device_id": device_id, "tablename": tn} for tn in tablenames
1895 ]
1896 # noinspection PyUnresolvedReferences
1897 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values) # type: ignore[attr-defined] # noqa: E501
1900def mark_all_tables_dirty(req: "CamcopsRequest") -> None:
1901 """
1902 If we are preserving, we assume that all tables are "dirty" (require work
1903 when we complete the upload) unless we specifically mark them clean.
1904 """
1905 device_id = req.tabletsession.device_id
1906 # Delete first
1907 # noinspection PyUnresolvedReferences
1908 req.dbsession.execute(
1909 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id) # type: ignore[attr-defined] # noqa: E501
1910 )
1911 # Now insert
1912 # https://docs.sqlalchemy.org/en/latest/core/tutorial.html#execute-multiple
1913 all_client_tablenames = list(CLIENT_TABLE_MAP.keys())
1914 insert_values = [
1915 {"device_id": device_id, "tablename": tn}
1916 for tn in all_client_tablenames
1917 ]
1918 # noinspection PyUnresolvedReferences
1919 req.dbsession.execute(DirtyTable.__table__.insert(), insert_values) # type: ignore[attr-defined] # noqa: E501
1922def mark_table_clean(req: "CamcopsRequest", table: Table) -> None:
1923 """
1924 Marks a table as being clean: that is,
1926 - the table has been scanned during the current upload
1927 - there is nothing to do (either from the current upload, OR A PREVIOUS
1928 UPLOAD).
1929 """
1930 tablename = table.name
1931 device_id = req.tabletsession.device_id
1932 # noinspection PyUnresolvedReferences
1933 req.dbsession.execute(
1934 DirtyTable.__table__.delete() # type: ignore[attr-defined]
1935 .where(DirtyTable.device_id == device_id)
1936 .where(DirtyTable.tablename == tablename)
1937 )
1940def mark_tables_clean(req: "CamcopsRequest", tables: List[Table]) -> None:
1941 """
1942 Marks multiple tables as clean.
1943 """
1944 if not tables:
1945 return
1946 device_id = req.tabletsession.device_id
1947 tablenames = [t.name for t in tables]
1948 # Delete first
1949 # noinspection PyUnresolvedReferences
1950 req.dbsession.execute(
1951 DirtyTable.__table__.delete() # type: ignore[attr-defined]
1952 .where(DirtyTable.device_id == device_id)
1953 .where(DirtyTable.tablename.in_(tablenames))
1954 )
1957def get_dirty_tables(req: "CamcopsRequest") -> List[Table]:
1958 """
1959 Returns tables marked as dirty for this device. (See
1960 :func:`mark_table_dirty`.)
1961 """
1962 query = select(DirtyTable.tablename).where(
1963 DirtyTable.device_id == req.tabletsession.device_id
1964 )
1965 tablenames = fetch_all_first_values(req.dbsession, query)
1966 return [CLIENT_TABLE_MAP[tn] for tn in tablenames]
1969def commit_all(req: "CamcopsRequest", batchdetails: BatchDetails) -> None:
1970 """
1971 Commits additions, removals, and preservations for all tables.
1973 Args:
1974 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
1975 batchdetails: the :class:`BatchDetails`
1976 """
1977 tables = get_dirty_tables(req)
1978 # log.debug("Dirty tables: {}", list(t.name for t in tables))
1979 tables.sort(key=upload_commit_order_sorter)
1981 changelist = [] # type: List[UploadTableChanges]
1982 for table in tables:
1983 auditinfo = commit_table(req, batchdetails, table, clear_dirty=False)
1984 changelist.append(auditinfo)
1986 if batchdetails.preserving:
1987 # Also preserve/finalize any corresponding special notes (2015-02-01),
1988 # but all in one go (2018-11-13).
1989 # noinspection PyUnresolvedReferences
1990 req.dbsession.execute(
1991 update(SpecialNote.__table__) # type: ignore[arg-type]
1992 .where(SpecialNote.device_id == req.tabletsession.device_id)
1993 .where(SpecialNote.era == ERA_NOW)
1994 .values(era=batchdetails.new_era)
1995 )
1997 clear_dirty_tables(req)
1998 audit_upload(req, changelist)
2000 # Performance 2018-11-13:
2001 # - start at 2.407 s
2002 # - remove final temptable clearance and COUNT(*): 1.626 to 2.118 s
2003 # - IN clause using Python literal not temptable: 1.18 to 1.905 s
2004 # - minor tidy: 1.075 to 1.65
2005 # - remove ORDER BY from task indexing: 1.093 to 1.607
2006 # - optimize special note code won't affect this: 1.076 to 1.617
2007 # At this point, entire upload process ~5s.
2008 # - big difference from commit_table() query optimization
2009 # - huge difference from being more careful with mark_table_dirty()
2010 # - further table scanning optimizations: fewer queries
2011 # Overall upload down to ~2.4s
2014def commit_table(
2015 req: "CamcopsRequest",
2016 batchdetails: BatchDetails,
2017 table: Table,
2018 clear_dirty: bool = True,
2019) -> UploadTableChanges:
2020 """
2021 Commits additions, removals, and preservations for one table.
2023 Should ONLY be called by :func:`commit_all`.
2025 Also updates task indexes.
2027 Args:
2028 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2029 batchdetails: the :class:`BatchDetails`
2030 table: SQLAlchemy :class:`Table`
2031 clear_dirty: remove the table from the record of dirty tables for
2032 this device? (If called from :func:`commit_all`, this should be
2033 ``False``, since it's faster to clear all dirty tables for the
2034 device simultaneously than one-by-one.)
2036 Returns:
2037 an :class:`UploadTableChanges` object
2038 """
2040 # Tried storing PKs in temporary tables, rather than using an IN clause
2041 # with Python values, as per
2042 # https://www.xaprb.com/blog/2006/06/28/why-large-in-clauses-are-problematic/ # noqa
2043 # However, it was slow.
2044 # We can gain a lot of efficiency (empirically) by:
2045 # - Storing PKs in Python
2046 # - Only performing updates when we need to
2047 # - Using a single query per table to get "add/remove/preserve" PKs
2049 # -------------------------------------------------------------------------
2050 # Helpful temporary variables
2051 # -------------------------------------------------------------------------
2052 user_id = req.user_id
2053 device_id = req.tabletsession.device_id
2054 exacttime = req.now
2055 dbsession = req.dbsession
2056 tablename = table.name
2057 batchtime = batchdetails.batchtime
2058 preserving = batchdetails.preserving
2060 # -------------------------------------------------------------------------
2061 # Fetch addition, removal, preservation, current PKs in a single query
2062 # -------------------------------------------------------------------------
2063 tablechanges = UploadTableChanges(table)
2064 serverrecs = get_server_live_records(
2065 req, device_id, table, current_only=False
2066 )
2067 for sr in serverrecs:
2068 tablechanges.note_serverrec(sr, preserving=preserving)
2070 # -------------------------------------------------------------------------
2071 # Additions
2072 # -------------------------------------------------------------------------
2073 # Update the records we're adding
2074 addition_pks = tablechanges.addition_pks
2075 if addition_pks:
2076 # log.debug("commit_table: {}, adding server PKs {}",
2077 # tablename, addition_pks)
2078 dbsession.execute(
2079 update(table)
2080 .where(table.c[FN_PK].in_(addition_pks))
2081 .values(
2082 {
2083 FN_CURRENT: 1,
2084 FN_ADDITION_PENDING: 0,
2085 FN_ADDING_USER_ID: user_id,
2086 FN_WHEN_ADDED_EXACT: exacttime,
2087 FN_WHEN_ADDED_BATCH_UTC: batchtime,
2088 }
2089 )
2090 )
2092 # -------------------------------------------------------------------------
2093 # Removals
2094 # -------------------------------------------------------------------------
2095 # Update the records we're removing
2096 removal_pks = tablechanges.removal_pks
2097 if removal_pks:
2098 # log.debug("commit_table: {}, removing server PKs {}",
2099 # tablename, removal_pks)
2100 dbsession.execute(
2101 update(table)
2102 .where(table.c[FN_PK].in_(removal_pks))
2103 .values(values_delete_now(req, batchdetails))
2104 )
2106 # -------------------------------------------------------------------------
2107 # Preservation
2108 # -------------------------------------------------------------------------
2109 # Preserve necessary records
2110 preservation_pks = tablechanges.preservation_pks
2111 if preservation_pks:
2112 # log.debug("commit_table: {}, preserving server PKs {}",
2113 # tablename, preservation_pks)
2114 new_era = batchdetails.new_era
2115 dbsession.execute(
2116 update(table)
2117 .where(table.c[FN_PK].in_(preservation_pks))
2118 .values(values_preserve_now(req, batchdetails))
2119 )
2120 if not preserving:
2121 # Also preserve/finalize any corresponding special notes
2122 # (2015-02-01), just for records being specifically preserved. If
2123 # we are preserving, this step happens in one go in commit_all()
2124 # (2018-11-13).
2125 # noinspection PyUnresolvedReferences
2126 dbsession.execute(
2127 update(SpecialNote.__table__) # type: ignore[arg-type]
2128 .where(SpecialNote.basetable == tablename)
2129 .where(SpecialNote.device_id == device_id)
2130 .where(SpecialNote.era == ERA_NOW)
2131 .where(
2132 exists()
2133 .select_from(table)
2134 .where(table.c[TABLET_ID_FIELD] == SpecialNote.task_id)
2135 .where(table.c[FN_DEVICE_ID] == SpecialNote.device_id)
2136 .where(table.c[FN_ERA] == new_era)
2137 )
2138 # ^^^^^^^^^^^^^^^^^^^^^^^^^^
2139 # This bit restricts to records being preserved.
2140 .values(era=new_era)
2141 )
2143 # -------------------------------------------------------------------------
2144 # Update special indexes
2145 # -------------------------------------------------------------------------
2146 update_indexes_and_push_exports(req, batchdetails, tablechanges)
2148 # -------------------------------------------------------------------------
2149 # Remove individually from list of dirty tables?
2150 # -------------------------------------------------------------------------
2151 if clear_dirty:
2152 # noinspection PyUnresolvedReferences
2153 dbsession.execute(
2154 DirtyTable.__table__.delete() # type: ignore[attr-defined]
2155 .where(DirtyTable.device_id == device_id)
2156 .where(DirtyTable.tablename == tablename)
2157 )
2158 # ... otherwise a call to clear_dirty_tables() must be made.
2160 if DEBUG_UPLOAD:
2161 log.debug("commit_table: {}", tablechanges)
2163 return tablechanges
2166def rollback_all(req: "CamcopsRequest") -> None:
2167 """
2168 Rolls back all pending changes for a device.
2169 """
2170 tables = get_dirty_tables(req)
2171 for table in tables:
2172 rollback_table(req, table)
2173 clear_dirty_tables(req)
2176def rollback_table(req: "CamcopsRequest", table: Table) -> None:
2177 """
2178 Rolls back changes for an individual table for a device.
2179 """
2180 device_id = req.tabletsession.device_id
2181 # Pending additions
2182 req.dbsession.execute(
2183 table.delete()
2184 .where(table.c[FN_DEVICE_ID] == device_id)
2185 .where(table.c[FN_ADDITION_PENDING])
2186 )
2187 # Pending deletions
2188 req.dbsession.execute(
2189 update(table)
2190 .where(table.c[FN_DEVICE_ID] == device_id)
2191 .where(table.c[FN_REMOVAL_PENDING])
2192 .values(
2193 {
2194 FN_REMOVAL_PENDING: 0,
2195 FN_WHEN_ADDED_EXACT: None,
2196 FN_WHEN_REMOVED_BATCH_UTC: None,
2197 FN_REMOVING_USER_ID: None,
2198 FN_SUCCESSOR_PK: None,
2199 }
2200 )
2201 )
2202 # Record-specific preservation (set by flag_record_for_preservation())
2203 req.dbsession.execute(
2204 update(table)
2205 .where(table.c[FN_DEVICE_ID] == device_id)
2206 .values({MOVE_OFF_TABLET_FIELD: 0})
2207 )
2210def clear_dirty_tables(req: "CamcopsRequest") -> None:
2211 """
2212 Clears the dirty-table list for a device.
2213 """
2214 device_id = req.tabletsession.device_id
2215 # noinspection PyUnresolvedReferences
2216 req.dbsession.execute(
2217 DirtyTable.__table__.delete().where(DirtyTable.device_id == device_id) # type: ignore[attr-defined] # noqa: E501
2218 )
2221# =============================================================================
2222# Additional helper functions for one-step upload
2223# =============================================================================
2226def process_table_for_onestep_upload(
2227 req: "CamcopsRequest",
2228 batchdetails: BatchDetails,
2229 table: Table,
2230 clientpk_name: str,
2231 rows: List[Dict[str, Any]],
2232) -> UploadTableChanges:
2233 """
2234 Performs all upload steps for a table.
2236 Note that we arrive here in a specific and safe table order; search for
2237 :func:`camcops_server.cc_modules.cc_client_api_helpers.upload_commit_order_sorter`.
2239 Args:
2240 req: the :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2241 batchdetails: the :class:`BatchDetails`
2242 table: an SQLAlchemy :class:`Table`
2243 clientpk_name: the name of the PK field on the client
2244 rows: a list of rows, where each row is a dictionary mapping field
2245 (column) names to values (those values being encoded as SQL-style
2246 literals in our extended syntax)
2248 Returns:
2249 an :class:`UploadTableChanges` object
2250 """
2251 serverrecs = get_server_live_records(
2252 req,
2253 req.tabletsession.device_id,
2254 table,
2255 clientpk_name,
2256 current_only=False,
2257 )
2258 servercurrentrecs = [r for r in serverrecs if r.current]
2259 if rows and not clientpk_name:
2260 fail_user_error(
2261 f"Client-side PK name not specified by client for "
2262 f"non-empty table {table.name!r}"
2263 )
2264 tablechanges = UploadTableChanges(table)
2265 server_pks_uploaded = [] # type: List[int]
2266 for row in rows:
2267 valuedict = {k: decode_single_value(v) for k, v in row.items()}
2268 urr = upload_record_core(
2269 req,
2270 batchdetails,
2271 table,
2272 clientpk_name,
2273 valuedict,
2274 server_live_current_records=servercurrentrecs,
2275 )
2276 # ... handles addition, modification, preservation, special processing
2277 # But we also make a note of these for indexing:
2278 if urr.oldserverpk is not None:
2279 server_pks_uploaded.append(urr.oldserverpk)
2280 tablechanges.note_urr(
2281 urr, preserving_new_records=batchdetails.preserving
2282 )
2283 # Which leaves:
2284 # (*) Deletion (where no record was uploaded at all)
2285 server_pks_for_deletion = [
2286 r.server_pk
2287 for r in servercurrentrecs
2288 if r.server_pk not in server_pks_uploaded
2289 ]
2290 if server_pks_for_deletion:
2291 flag_deleted(req, batchdetails, table, server_pks_for_deletion)
2292 tablechanges.note_removal_deleted_pks(server_pks_for_deletion)
2294 # Preserving all records not specifically processed above, too
2295 if batchdetails.preserving:
2296 # Preserve all, including noncurrent:
2297 preserve_all(req, batchdetails, table)
2298 # Note other preserved records, for indexing:
2299 tablechanges.note_preservation_pks(r.server_pk for r in serverrecs)
2301 # (*) Indexing (and push exports)
2302 update_indexes_and_push_exports(req, batchdetails, tablechanges)
2304 if DEBUG_UPLOAD:
2305 log.debug("process_table_for_onestep_upload: {}", tablechanges)
2307 return tablechanges
2310# =============================================================================
2311# Audit functions
2312# =============================================================================
2315def audit(
2316 req: "CamcopsRequest",
2317 details: str,
2318 patient_server_pk: int = None,
2319 tablename: str = None,
2320 server_pk: int = None,
2321) -> None:
2322 """
2323 Audit something.
2324 """
2325 # Add parameters and pass on:
2326 cc_audit.audit(
2327 req=req,
2328 details=details,
2329 patient_server_pk=patient_server_pk,
2330 table=tablename,
2331 server_pk=server_pk,
2332 device_id=req.tabletsession.device_id, # added
2333 remote_addr=req.remote_addr, # added
2334 user_id=req.user_id, # added
2335 from_console=False, # added
2336 from_dbclient=True, # added
2337 )
2340# =============================================================================
2341# Helper functions for single-user mode
2342# =============================================================================
2345def make_single_user_mode_username(
2346 client_device_name: str, patient_pk: int
2347) -> str:
2348 """
2349 Returns the username for single-user mode.
2350 """
2351 return f"user-{client_device_name}-{patient_pk}"
2354def json_patient_info(patient: Patient) -> str:
2355 """
2356 Converts patient details to a string representation of a JSON list (one
2357 patient) containing a single JSON dictionary (detailing that patient), with
2358 keys/formats known to the client.
2360 (One item list to be consistent with patients uploaded from the tablet.)
2362 Args:
2363 patient: :class:`camcops_server.cc_modules.cc_patient.Patient`
2364 """
2365 patient_dict = {
2366 TabletParam.SURNAME: patient.surname,
2367 TabletParam.FORENAME: patient.forename,
2368 TabletParam.SEX: patient.sex,
2369 TabletParam.DOB: format_datetime(
2370 patient.dob, DateFormat.ISO8601_DATE_ONLY
2371 ),
2372 TabletParam.EMAIL: patient.email,
2373 TabletParam.ADDRESS: patient.address,
2374 TabletParam.GP: patient.gp,
2375 TabletParam.OTHER: patient.other,
2376 }
2377 for idnum in patient.idnums:
2378 key = f"{TabletParam.IDNUM_PREFIX}{idnum.which_idnum}"
2379 patient_dict[key] = idnum.idnum_value # type: ignore[assignment]
2380 # One item list to be consistent with patients uploaded from the tablet
2381 return json.dumps([patient_dict])
2384def get_single_server_patient(req: "CamcopsRequest") -> Patient:
2385 """
2386 Returns the patient identified by the proquint access key present in this
2387 request, or raises.
2388 """
2389 _ = req.gettext
2391 patient_proquint = get_str_var(req, TabletParam.PATIENT_PROQUINT)
2392 assert patient_proquint is not None # For type checker
2394 try:
2395 uuid_obj = uuid_from_proquint(patient_proquint)
2396 except InvalidProquintException:
2397 # Checksum failed or characters in wrong place
2398 # We'll do the same validation on the client so in theory
2399 # should never get here
2400 fail_user_error(
2401 _(
2402 "There is no patient with access key '{access_key}'. "
2403 "Have you entered the key correctly?"
2404 ).format(access_key=patient_proquint)
2405 )
2407 server_device = Device.get_server_device(req.dbsession)
2409 # noinspection PyUnboundLocalVariable,PyProtectedMember
2410 patient = (
2411 req.dbsession.query(Patient)
2412 .filter(
2413 Patient.uuid == uuid_obj,
2414 Patient._device_id == server_device.id,
2415 Patient._era == ERA_NOW,
2416 Patient._current == True, # noqa: E712
2417 )
2418 .options(joinedload(Patient.task_schedules)) # type: ignore[arg-type]
2419 .one_or_none()
2420 )
2422 if patient is None:
2423 fail_user_error(
2424 _(
2425 "There is no patient with access key '{access_key}'. "
2426 "Have you entered the key correctly?"
2427 ).format(access_key=patient_proquint)
2428 )
2430 if not patient.idnums:
2431 # In theory should never happen. The patient must be created with at
2432 # least one ID number. We did see this once in testing (possibly when
2433 # a patient created on a device was registered)
2434 _ = req.gettext
2435 fail_server_error(_("Patient has no ID numbers"))
2437 return patient
2440def get_or_create_single_user(
2441 req: "CamcopsRequest", name: str, patient: Patient
2442) -> Tuple[User, str]:
2443 """
2444 Creates a user for a patient (who's using single-user mode).
2446 The user is associated (via its name) with the combination of a client
2447 device and a patient. (If a device is re-registered to another patient, the
2448 username will change.)
2450 If the username already exists, then since we can't look up the password
2451 (it's irreversibly encrypted), we will set it afresh.
2453 - Why is a user associated with a patient? So we can enforce that the user
2454 can upload only data relating to that patient.
2456 - Why is a user associated with a device?
2458 - If it is: then two users (e.g. "Device1-Bob" and "Device2-Bob") can
2459 independently work with the same patient. This will be highly
2460 confusing (mainly because it will allow "double" copies of tasks to be
2461 created, though only by manually entering things twice).
2463 - If it isn't (e.g. user "Bob"): then, because registering the patient on
2464 Device2 will reset the password for the user, registering a new device
2465 for a patient will "take over" from a previous device. That has some
2466 potential for data loss if work was in progress (incomplete tasks won't
2467 be uploadable any more, and re-registering [to fix the password on the
2468 first device] would delete data).
2470 - Since some confusion is better than some data loss, we associate users
2471 with a device/patient combination.
2473 Args:
2474 req:
2475 a :class:`camcops_server.cc_modules.cc_request.CamcopsRequest`
2476 name:
2477 username
2478 patient:
2479 associated :class:`camcops_server.cc_modules.cc_patient.Patient`,
2480 which also tells us the group in which to place this user
2482 Returns:
2483 tuple: :class:`camcops_server.cc_modules.cc_user.User`, password
2485 """
2486 dbsession = req.dbsession
2487 password = random_password()
2488 group = patient.group
2489 assert group is not None # for type checker
2491 user = User.get_user_by_name(dbsession, name)
2492 creating_new_user = user is None
2493 if creating_new_user:
2494 # Create a fresh user.
2495 user = User(username=name)
2496 user.upload_group = group
2497 user.auto_generated = True
2498 user.superuser = False # should be redundant!
2499 # noinspection PyProtectedMember
2500 user.single_patient_pk = patient._pk
2501 user.set_password(req, password)
2502 if creating_new_user:
2503 dbsession.add(user)
2504 # As the username is based on a UUID, we're pretty sure another
2505 # request won't have created the same user, otherwise we'd need
2506 # to catch IntegrityError
2507 dbsession.flush()
2509 membership = UserGroupMembership(user_id=user.id, group_id=group.id)
2510 membership.may_register_devices = True
2511 membership.may_upload = True
2512 user.user_group_memberships = [membership] # ... only these permissions
2514 return user, password
2517def random_password(length: int = 32) -> str:
2518 """
2519 Create a random password.
2520 """
2521 # Not trying anything clever with distributions of letters, digits etc
2522 characters = string.ascii_letters + string.digits + string.punctuation
2523 # We use secrets.choice() rather than random.choices() as it's better
2524 # for security/cryptography purposes.
2525 return "".join(secrets.choice(characters) for _ in range(length))
2528def get_task_schedules(req: "CamcopsRequest", patient: Patient) -> str:
2529 """
2530 Gets a JSON string representation of the task schedules for a specified
2531 patient.
2532 """
2533 dbsession = req.dbsession
2535 schedules = []
2537 for pts in patient.task_schedules:
2538 if pts.start_datetime is None:
2539 # Minutes granularity so we are consistent with the form
2540 pts.start_datetime = req.now_utc.replace(second=0, microsecond=0)
2541 dbsession.add(pts)
2543 items = []
2545 for task_info in pts.get_list_of_scheduled_tasks(req):
2546 due_from = task_info.start_datetime.to_iso8601_string()
2547 due_by = task_info.end_datetime.to_iso8601_string()
2549 complete = False
2550 when_completed = None
2551 task = task_info.task
2552 if task:
2553 complete = task.is_complete()
2554 if complete and task.when_last_modified:
2555 when_completed = (
2556 task.when_last_modified.to_iso8601_string()
2557 )
2559 if pts.settings is not None:
2560 settings = pts.settings.get(task_info.tablename, {})
2561 else:
2562 settings = {}
2564 items.append(
2565 {
2566 TabletParam.TABLE: task_info.tablename,
2567 TabletParam.ANONYMOUS: task_info.is_anonymous,
2568 TabletParam.SETTINGS: settings,
2569 TabletParam.DUE_FROM: due_from,
2570 TabletParam.DUE_BY: due_by,
2571 TabletParam.COMPLETE: complete,
2572 TabletParam.WHEN_COMPLETED: when_completed,
2573 }
2574 )
2576 schedules.append(
2577 {
2578 TabletParam.TASK_SCHEDULE_NAME: pts.task_schedule.name,
2579 TabletParam.TASK_SCHEDULE_ITEMS: items,
2580 }
2581 )
2583 return json.dumps(schedules)
2586# =============================================================================
2587# Action processors: allowed to any user
2588# =============================================================================
2589# If they return None, the framework uses the operation name as the reply in
2590# the success message. Not returning anything is the same as returning None.
2591# Authentication is performed in advance of these.
2594def op_check_device_registered(req: "CamcopsRequest") -> None:
2595 """
2596 Check that a device is registered, or raise
2597 :exc:`UserErrorException`.
2598 """
2599 req.tabletsession.ensure_device_registered()
2602def op_register_patient(req: "CamcopsRequest") -> Dict[str, Any]:
2603 """
2604 Registers a patient. That is, the client provides an access key. If all
2605 is well, the server returns details of that patient, as well as key
2606 server parameters, plus (if required) the username/password to use.
2607 """
2608 # -------------------------------------------------------------------------
2609 # Patient details
2610 # -------------------------------------------------------------------------
2611 patient = get_single_server_patient(req) # may fail/raise
2612 patient_info = json_patient_info(patient)
2613 reply_dict = {TabletParam.PATIENT_INFO: patient_info}
2615 # -------------------------------------------------------------------------
2616 # Username/password
2617 # -------------------------------------------------------------------------
2618 client_device_name = get_str_var(req, TabletParam.DEVICE)
2619 # noinspection PyProtectedMember
2620 user_name = make_single_user_mode_username(client_device_name, patient._pk)
2621 user, password = get_or_create_single_user(req, user_name, patient)
2622 reply_dict[TabletParam.USER] = user.username
2623 reply_dict[TabletParam.PASSWORD] = password
2625 # -------------------------------------------------------------------------
2626 # Intellectual property settings
2627 # -------------------------------------------------------------------------
2628 ip_use = patient.group.ip_use or IpUse()
2629 # ... if the group doesn't have an associated ip_use object, use defaults
2630 ip_dict = {
2631 TabletParam.IP_USE_COMMERCIAL: int(ip_use.commercial),
2632 TabletParam.IP_USE_CLINICAL: int(ip_use.clinical),
2633 TabletParam.IP_USE_EDUCATIONAL: int(ip_use.educational),
2634 TabletParam.IP_USE_RESEARCH: int(ip_use.research),
2635 }
2636 reply_dict[TabletParam.IP_USE_INFO] = json.dumps(ip_dict)
2638 return reply_dict
2641# =============================================================================
2642# Action processors that require REGISTRATION privilege
2643# =============================================================================
2646def op_register_device(req: "CamcopsRequest") -> Dict[str, Any]:
2647 """
2648 Register a device with the server.
2650 Returns:
2651 server information dictionary (from :func:`get_server_id_info`)
2652 """
2653 dbsession = req.dbsession
2654 ts = req.tabletsession
2655 device_friendly_name = get_str_var(
2656 req, TabletParam.DEVICE_FRIENDLY_NAME, mandatory=False
2657 )
2658 # noinspection PyUnresolvedReferences
2659 device_exists = exists_in_table(
2660 dbsession, Device.__table__, Device.name == ts.device_name # type: ignore[arg-type] # noqa: E501
2661 )
2662 if device_exists:
2663 # device already registered, but accept re-registration
2664 # noinspection PyUnresolvedReferences
2665 dbsession.execute(
2666 update(Device.__table__) # type: ignore[arg-type]
2667 .where(Device.name == ts.device_name)
2668 .values(
2669 friendly_name=device_friendly_name,
2670 camcops_version=ts.tablet_version_str,
2671 registered_by_user_id=req.user_id,
2672 when_registered_utc=req.now_utc,
2673 )
2674 )
2675 else:
2676 # new registration
2677 try:
2678 # noinspection PyUnresolvedReferences
2679 dbsession.execute(
2680 Device.__table__.insert().values( # type: ignore[attr-defined]
2681 name=ts.device_name,
2682 friendly_name=device_friendly_name,
2683 camcops_version=ts.tablet_version_str,
2684 registered_by_user_id=req.user_id,
2685 when_registered_utc=req.now_utc,
2686 )
2687 )
2688 except IntegrityError:
2689 fail_user_error(INSERT_FAILED)
2691 ts.reload_device()
2692 audit(
2693 req,
2694 f"register, device_id={ts.device_id}, "
2695 f"friendly_name={device_friendly_name}",
2696 tablename=Device.__tablename__,
2697 )
2698 return get_server_id_info(req)
2701def op_get_extra_strings(req: "CamcopsRequest") -> Dict[str, str]:
2702 """
2703 Fetch all local extra strings from the server.
2705 Returns:
2706 a SELECT-style reply (see :func:`get_select_reply`) for the
2707 extra-string table
2708 """
2709 fields = [
2710 ExtraStringFieldNames.TASK,
2711 ExtraStringFieldNames.NAME,
2712 ExtraStringFieldNames.LANGUAGE,
2713 ExtraStringFieldNames.VALUE,
2714 ]
2715 rows = req.get_all_extra_strings()
2716 reply = get_select_reply(fields, rows)
2717 audit(req, "get_extra_strings")
2718 return reply
2721# noinspection PyUnusedLocal
2722def op_get_allowed_tables(req: "CamcopsRequest") -> Dict[str, str]:
2723 """
2724 Returns the names of all possible tables on the server, each paired with
2725 the minimum client (tablet) version that will be accepted for each table.
2726 (Typically these are all the same as the minimum global tablet version.)
2728 Uses the SELECT-like syntax (see :func:`get_select_reply`).
2729 """
2730 tables_versions = all_tables_with_min_client_version()
2731 fields = [
2732 AllowedTablesFieldNames.TABLENAME,
2733 AllowedTablesFieldNames.MIN_CLIENT_VERSION,
2734 ]
2735 rows = [[k, str(v)] for k, v in tables_versions.items()]
2736 reply = get_select_reply(fields, rows)
2737 audit(req, "get_allowed_tables")
2738 return reply
2741def op_get_task_schedules(req: "CamcopsRequest") -> Dict[str, str]:
2742 """
2743 Return details of the task schedules for the patient associated with
2744 this request, for single-user mode. Also returns details of the single
2745 patient, in case that's changed.
2746 """
2747 patient = get_single_server_patient(req)
2748 patient_info = json_patient_info(patient)
2749 task_schedules = get_task_schedules(req, patient)
2750 return {
2751 TabletParam.PATIENT_INFO: patient_info,
2752 TabletParam.TASK_SCHEDULES: task_schedules,
2753 }
2756# =============================================================================
2757# Action processors that require UPLOAD privilege
2758# =============================================================================
2761# noinspection PyUnusedLocal
2762def op_check_upload_user_and_device(req: "CamcopsRequest") -> None:
2763 """
2764 Stub function for the operation to check that a user is valid.
2766 To get this far, the user has to be valid, so this function doesn't
2767 actually have to do anything.
2768 """
2769 pass # don't need to do anything!
2772# noinspection PyUnusedLocal
2773def op_get_id_info(req: "CamcopsRequest") -> Dict[str, Any]:
2774 """
2775 Fetch server ID information; see :func:`get_server_id_info`.
2776 """
2777 return get_server_id_info(req)
2780def op_start_upload(req: "CamcopsRequest") -> None:
2781 """
2782 Begin an upload.
2783 """
2784 start_device_upload_batch(req)
2787def op_end_upload(req: "CamcopsRequest") -> None:
2788 """
2789 Ends an upload and commits changes.
2790 """
2791 batchdetails = get_batch_details(req)
2792 # ensure it's the same user finishing as starting!
2793 end_device_upload_batch(req, batchdetails)
2796def op_upload_table(req: "CamcopsRequest") -> str:
2797 """
2798 Upload a table.
2800 Incoming information in the POST request includes a CSV list of fields, a
2801 count of the number of records being provided, and a set of variables named
2802 ``record0`` ... ``record{nrecords - 1}``, each containing a CSV list of
2803 SQL-encoded values.
2805 Typically used for smaller tables, i.e. most except for BLOBs.
2806 """
2807 table = get_table_from_req(req, TabletParam.TABLE)
2809 allowed_nonexistent_fields = [] # type: List[str]
2810 # noinspection PyUnresolvedReferences
2811 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:
2812 for x in range(1, NUMBER_OF_IDNUMS_DEFUNCT + 1):
2813 allowed_nonexistent_fields.extend(
2814 [
2815 FP_ID_NUM + str(x),
2816 FP_ID_DESC + str(x),
2817 FP_ID_SHORT_DESC + str(x),
2818 ]
2819 )
2821 fields = get_fields_from_post_var(
2822 req,
2823 table,
2824 TabletParam.FIELDS,
2825 allowed_nonexistent_fields=allowed_nonexistent_fields,
2826 )
2827 nrecords = get_int_var(req, TabletParam.NRECORDS)
2829 nfields = len(fields)
2830 if nfields < 1:
2831 # May never be reached as the POST var can't be empty
2832 fail_user_error(
2833 f"{TabletParam.FIELDS}={nfields}: can't be less than 1"
2834 )
2835 if nrecords < 0:
2836 fail_user_error(
2837 f"{TabletParam.NRECORDS}={nrecords}: can't be less than 0"
2838 )
2840 batchdetails = get_batch_details(req)
2842 ts = req.tabletsession
2843 if ts.explicit_pkname_for_upload_table: # q.v.
2844 # New client: tells us the PK name explicitly.
2845 clientpk_name = get_single_field_from_post_var(
2846 req, table, TabletParam.PKNAME
2847 )
2848 else:
2849 # Old client. Either (a) old Titanium client, in which the client PK
2850 # is in fields[0] [SUCH CLIENTS ARE NO LONGER SUPPORTED AS OF SERVER
2851 # v2.4.15], or (b) an early C++ client, in which there was no
2852 # guaranteed order (and no explicit PK name was sent). However, in
2853 # either case, the client PK name was (is) always "id".
2854 clientpk_name = TABLET_ID_FIELD
2855 ensure_valid_field_name(table, clientpk_name)
2856 server_pks_uploaded = [] # type: List[int]
2857 n_new = 0
2858 n_modified = 0
2859 n_identical = 0
2860 dirty = False
2861 serverrecs = get_server_live_records(
2862 req,
2863 ts.device_id,
2864 table,
2865 clientpk_name=clientpk_name,
2866 current_only=True,
2867 )
2868 for r in range(nrecords):
2869 recname = TabletParam.RECORD_PREFIX + str(r)
2870 values = get_values_from_post_var(req, recname)
2871 nvalues = len(values)
2872 if nvalues != nfields:
2873 errmsg = (
2874 f"Number of fields in field list ({nfields}) doesn't match "
2875 f"number of values in record {r} ({nvalues})"
2876 )
2877 log.warning(errmsg + f"\nfields: {fields!r}\nvalues: {values!r}")
2878 fail_user_error(errmsg)
2879 valuedict = dict(zip(fields, values))
2880 # log.debug("table {!r}, record {}: {!r}", table.name, r, valuedict)
2881 # CORE: CALLS upload_record_core
2882 urr = upload_record_core(
2883 req,
2884 batchdetails,
2885 table,
2886 clientpk_name,
2887 valuedict,
2888 server_live_current_records=serverrecs,
2889 )
2890 if urr.oldserverpk is not None: # was an existing record
2891 server_pks_uploaded.append(urr.oldserverpk)
2892 if urr.newserverpk is None:
2893 n_identical += 1
2894 else:
2895 n_modified += 1
2896 else: # entirely new
2897 n_new += 1
2898 if urr.dirty:
2899 dirty = True
2901 # Now deal with any ABSENT (not in uploaded data set) conditions.
2902 server_pks_for_deletion = [
2903 r.server_pk
2904 for r in serverrecs
2905 if r.server_pk not in server_pks_uploaded
2906 ]
2907 # Note that "deletion" means "end of the line"; records that are modified
2908 # and replaced were handled by upload_record_core().
2909 n_deleted = len(server_pks_for_deletion)
2910 if n_deleted > 0:
2911 flag_deleted(req, batchdetails, table, server_pks_for_deletion)
2913 # Set dirty/clean status
2914 if (
2915 dirty
2916 or n_new > 0
2917 or n_modified > 0
2918 or n_deleted > 0
2919 or any(sr.move_off_tablet for sr in serverrecs)
2920 ):
2921 # ... checks on n_new and n_modified are redundant; dirty will be True
2922 mark_table_dirty(req, table)
2923 elif batchdetails.preserving and not serverrecs:
2924 # We've scanned this table, and there would be no work to do to
2925 # preserve records from previous uploads.
2926 mark_table_clean(req, table)
2928 # Special for old tablets:
2929 # noinspection PyUnresolvedReferences
2930 if req.tabletsession.cope_with_old_idnums and table == Patient.__table__:
2931 # noinspection PyUnresolvedReferences
2932 mark_table_dirty(req, PatientIdNum.__table__) # type: ignore[arg-type]
2933 # Mark patient ID numbers for deletion if their parent Patient is
2934 # similarly being marked for deletion
2935 # noinspection PyUnresolvedReferences,PyProtectedMember
2936 req.dbsession.execute(
2937 update(PatientIdNum.__table__) # type: ignore[arg-type]
2938 .where(PatientIdNum._device_id == Patient._device_id)
2939 .where(PatientIdNum._era == ERA_NOW)
2940 .where(PatientIdNum.patient_id == Patient.id)
2941 .where(Patient._pk.in_(server_pks_for_deletion))
2942 .where(Patient._era == ERA_NOW) # shouldn't be in doubt!
2943 .values(_removal_pending=1, _successor_pk=None)
2944 )
2946 # Auditing occurs at commit_all.
2947 log.info(
2948 "Upload successful; {n} records uploaded to table {t} "
2949 "({new} new, {mod} modified, {i} identical, {nd} deleted)",
2950 n=nrecords,
2951 t=table.name,
2952 new=n_new,
2953 mod=n_modified,
2954 i=n_identical,
2955 nd=n_deleted,
2956 )
2957 return f"Table {table.name} upload successful"
2960def op_upload_record(req: "CamcopsRequest") -> str:
2961 """
2962 Upload an individual record. (Typically used for BLOBs.)
2963 Incoming POST information includes a CSV list of fields and a CSV list of
2964 values.
2965 """
2966 batchdetails = get_batch_details(req)
2967 table = get_table_from_req(req, TabletParam.TABLE)
2968 clientpk_name = get_single_field_from_post_var(
2969 req, table, TabletParam.PKNAME
2970 )
2971 valuedict = get_fields_and_values(
2972 req, table, TabletParam.FIELDS, TabletParam.VALUES
2973 )
2974 urr = upload_record_core(
2975 req, batchdetails, table, clientpk_name, valuedict
2976 )
2977 if urr.dirty:
2978 mark_table_dirty(req, table)
2979 if urr.oldserverpk is None:
2980 log.info("upload-insert")
2981 return "UPLOAD-INSERT"
2982 else:
2983 if urr.newserverpk is None:
2984 log.info("upload-update: skipping existing record")
2985 else:
2986 log.info("upload-update")
2987 return "UPLOAD-UPDATE"
2988 # Auditing occurs at commit_all.
2991def op_upload_empty_tables(req: "CamcopsRequest") -> str:
2992 """
2993 The tablet supplies a list of tables that are empty at its end, and we
2994 will 'wipe' all appropriate tables; this reduces the number of HTTP
2995 requests.
2996 """
2997 tables = get_tables_from_post_var(req, TabletParam.TABLES)
2998 batchdetails = get_batch_details(req)
2999 to_dirty = [] # type: List[Table]
3000 to_clean = [] # type: List[Table]
3001 for table in tables:
3002 nrows_affected = flag_all_records_deleted(req, table)
3003 if nrows_affected > 0:
3004 to_dirty.append(table)
3005 elif batchdetails.preserving:
3006 # There are no records in the current era for this device.
3007 to_clean.append(table)
3008 # In the fewest number of queries:
3009 mark_tables_dirty(req, to_dirty)
3010 mark_tables_clean(req, to_clean)
3011 log.info("upload_empty_tables")
3012 # Auditing occurs at commit_all.
3013 return "UPLOAD-EMPTY-TABLES"
3016def op_start_preservation(req: "CamcopsRequest") -> str:
3017 """
3018 Marks this upload batch as one in which all records will be preserved
3019 (i.e. moved from NOW-era to an older era, so they can be deleted safely
3020 from the tablet).
3022 Without this, individual records can still be marked for preservation if
3023 their MOVE_OFF_TABLET_FIELD field (``_move_off_tablet``) is set; see
3024 :func:`upload_record` and the functions it calls.
3025 """
3026 get_batch_details(req)
3027 start_preserving(req)
3028 log.info("start_preservation successful")
3029 # Auditing occurs at commit_all.
3030 return "STARTPRESERVATION"
3033def op_delete_where_key_not(req: "CamcopsRequest") -> str:
3034 """
3035 Marks records for deletion, for a device/table, where the client PK
3036 is not in a specified list.
3037 """
3038 table = get_table_from_req(req, TabletParam.TABLE)
3039 clientpk_name = get_single_field_from_post_var(
3040 req, table, TabletParam.PKNAME
3041 )
3042 clientpk_values = get_values_from_post_var(req, TabletParam.PKVALUES)
3044 get_batch_details(req)
3045 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)
3046 # Auditing occurs at commit_all.
3047 # log.info("delete_where_key_not successful; table {} trimmed", table)
3048 return "Trimmed"
3051def op_which_keys_to_send(req: "CamcopsRequest") -> str:
3052 """
3053 Intended use: "For my device, and a specified table, here are my
3054 client-side PKs (as a CSV list), and the modification dates for each
3055 corresponding record (as a CSV list). Please tell me which records have
3056 mismatching dates on the server, i.e. those that I need to re-upload."
3058 Used particularly for BLOBs, to reduce traffic, i.e. so we don't have to
3059 send a lot of BLOBs.
3061 Note new ``TabletParam.MOVE_OFF_TABLET_VALUES`` parameter in server v2.3.0,
3062 with bugfix for pre-2.3.0 clients that won't send this; see changelog.
3063 """
3064 # -------------------------------------------------------------------------
3065 # Get details
3066 # -------------------------------------------------------------------------
3067 table = get_table_from_req(req, TabletParam.TABLE)
3068 clientpk_name = get_single_field_from_post_var(
3069 req, table, TabletParam.PKNAME
3070 )
3071 clientpk_values = get_values_from_post_var(
3072 req, TabletParam.PKVALUES, mandatory=False
3073 )
3074 # ... should be autoconverted to int, but we check below
3075 client_dates = get_values_from_post_var(
3076 req, TabletParam.DATEVALUES, mandatory=False
3077 )
3078 # ... will be in string format
3080 npkvalues = len(clientpk_values)
3081 ndatevalues = len(client_dates)
3082 if npkvalues != ndatevalues:
3083 fail_user_error(
3084 f"Number of PK values ({npkvalues}) doesn't match number of dates "
3085 f"({ndatevalues})"
3086 )
3088 # v2.3.0:
3089 move_off_tablet_values = [] # type: List[int] # for type checker
3090 if req.has_param(TabletParam.MOVE_OFF_TABLET_VALUES):
3091 client_reports_move_off_tablet = True
3092 move_off_tablet_values = get_values_from_post_var(
3093 req, TabletParam.MOVE_OFF_TABLET_VALUES, mandatory=True
3094 )
3095 # ... should be autoconverted to int
3096 n_motv = len(move_off_tablet_values)
3097 if n_motv != npkvalues:
3098 fail_user_error(
3099 f"Number of move-off-tablet values ({n_motv}) doesn't match "
3100 f"number of PKs ({npkvalues})"
3101 )
3102 try:
3103 move_off_tablet_values = [bool(x) for x in move_off_tablet_values]
3104 except (TypeError, ValueError):
3105 # Probably never reached given pretty much anything standard can be
3106 # converted to bool
3107 fail_user_error(
3108 f"Bad move-off-tablet values: {move_off_tablet_values!r}"
3109 )
3110 else:
3111 client_reports_move_off_tablet = False
3112 log.warning(
3113 "op_which_keys_to_send: old client not reporting "
3114 "{}; requesting all records",
3115 TabletParam.MOVE_OFF_TABLET_VALUES,
3116 )
3118 clientinfo = [] # type: List[WhichKeyToSendInfo]
3120 for i in range(npkvalues):
3121 cpkv = clientpk_values[i]
3122 if not isinstance(cpkv, int):
3123 fail_user_error(f"Bad (non-integer) client PK: {cpkv!r}")
3124 try:
3125 dt = coerce_to_pendulum(client_dates[i])
3126 if dt is None:
3127 fail_user_error(f"Missing date/time for client PK {cpkv}")
3128 except ValueError:
3129 fail_user_error(f"Bad date/time: {client_dates[i]!r}")
3130 clientinfo.append(
3131 WhichKeyToSendInfo(
3132 client_pk=cpkv,
3133 client_when=dt,
3134 client_move_off_tablet=(
3135 move_off_tablet_values[i] # type: ignore[arg-type]
3136 if client_reports_move_off_tablet
3137 else False
3138 ),
3139 )
3140 )
3142 # -------------------------------------------------------------------------
3143 # Work out the answer
3144 # -------------------------------------------------------------------------
3145 batchdetails = get_batch_details(req)
3147 # 1. The client sends us all its PKs. So "delete" anything not in that
3148 # list.
3149 flag_deleted_where_clientpk_not(req, table, clientpk_name, clientpk_values)
3151 # 2. See which ones are new or updates.
3152 client_pks_needed = [] # type: List[int]
3153 client_pk_to_serverrec = client_pks_that_exist(
3154 req, table, clientpk_name, clientpk_values
3155 )
3156 for wk in clientinfo:
3157 if client_reports_move_off_tablet:
3158 if wk.client_pk not in client_pk_to_serverrec:
3159 # New on the client; we want it
3160 client_pks_needed.append(wk.client_pk)
3161 else:
3162 # We know about some version of this client record.
3163 serverrec = client_pk_to_serverrec[wk.client_pk]
3164 if serverrec.server_when != wk.client_when:
3165 # Modified on the client; we want it
3166 client_pks_needed.append(wk.client_pk)
3167 elif serverrec.move_off_tablet != wk.client_move_off_tablet:
3168 # Not modified on the client. But it is being preserved.
3169 # We don't need to ask the client for it again, but we do
3170 # need to mark the preservation.
3171 flag_record_for_preservation(
3172 req, batchdetails, table, serverrec.server_pk
3173 )
3175 else:
3176 # Client hasn't told us about the _move_off_tablet flag. Always
3177 # request the record (workaround potential bug in old clients).
3178 client_pks_needed.append(wk.client_pk)
3180 # Success
3181 pk_csv_list = ",".join(
3182 [str(x) for x in client_pks_needed if x is not None]
3183 )
3184 # log.info("which_keys_to_send successful: table {}", table.name)
3185 return pk_csv_list
3188def op_validate_patients(req: "CamcopsRequest") -> str:
3189 """
3190 As of v2.3.0, the client can use this command to validate patients against
3191 arbitrary server criteria -- definitely the upload/finalize ID policies,
3192 but potentially also other criteria of the server's (like matching against
3193 a bank of predefined patients).
3195 Compare ``NetworkManager::getPatientInfoJson()`` on the client.
3197 There is a slight weakness with respect to "single-patient" users, in that
3198 the client *asks* if the patients are OK (rather than the server
3199 *enforcing* that they are OK, via hooks into :func:`op_upload_table`,
3200 :func:`op_upload_record`, :func:`op_upload_entire_database` -- made more
3201 complex because ID numbers are not uploaded to the same table...). In
3202 principle, the weakness is that a user could (a) crack their assigned
3203 password and (b) rework the CamCOPS client, in order to upload "bad"
3204 patient data into their assigned group.
3206 todo:
3207 address this by having the server *require* patient validation for
3208 all uploads?
3210 """
3211 pt_json_list = get_json_from_post_var(
3212 req,
3213 TabletParam.PATIENT_INFO,
3214 decoder=PATIENT_INFO_JSON_DECODER,
3215 mandatory=True,
3216 )
3217 if not isinstance(pt_json_list, list):
3218 fail_user_error("Top-level JSON is not a list")
3219 group = Group.get_group_by_id(req.dbsession, req.user.upload_group_id)
3220 for pt_dict in pt_json_list:
3221 ensure_valid_patient_json(req, group, pt_dict)
3222 return SUCCESS_MSG
3225def op_upload_entire_database(req: "CamcopsRequest") -> str:
3226 """
3227 Perform a one-step upload of the entire database.
3229 - From v2.3.0.
3230 - Therefore, we do not have to cope with old-style ID numbers.
3231 """
3232 # Roll back and clear any outstanding changes
3233 clear_device_upload_batch(req)
3235 # Fetch the JSON, with sanity checks
3236 preserving = get_bool_int_var(req, TabletParam.FINALIZING)
3237 pknameinfo = get_json_from_post_var(
3238 req, TabletParam.PKNAMEINFO, decoder=DB_JSON_DECODER, mandatory=True
3239 )
3240 if not isinstance(pknameinfo, dict):
3241 fail_user_error("PK name info JSON is not a dict")
3242 dbdata = get_json_from_post_var(
3243 req, TabletParam.DBDATA, decoder=DB_JSON_DECODER, mandatory=True
3244 )
3245 if not isinstance(dbdata, dict):
3246 fail_user_error("Database data JSON is not a dict")
3248 # Sanity checks
3249 dbdata_tablenames = sorted(dbdata.keys())
3250 pkinfo_tablenames = sorted(pknameinfo.keys())
3251 if pkinfo_tablenames != dbdata_tablenames:
3252 fail_user_error("Table names don't match from (1) DB data (2) PK info")
3253 duff_tablenames = sorted(
3254 list(set(dbdata_tablenames) - set(CLIENT_TABLE_MAP.keys()))
3255 )
3256 if duff_tablenames:
3257 fail_user_error(
3258 f"Attempt to upload nonexistent tables: {duff_tablenames!r}"
3259 )
3261 # Perform the upload
3262 batchdetails = BatchDetails(
3263 req.now_utc, preserving=preserving, onestep=True
3264 ) # NB special "onestep" option
3265 # Process the tables in a certain order:
3266 tables = sorted(CLIENT_TABLE_MAP.values(), key=upload_commit_order_sorter)
3267 changelist = [] # type: List[UploadTableChanges]
3268 for table in tables:
3269 clientpk_name = pknameinfo.get(table.name, "")
3270 rows = dbdata.get(table.name, [])
3271 tablechanges = process_table_for_onestep_upload(
3272 req, batchdetails, table, clientpk_name, rows
3273 )
3274 changelist.append(tablechanges)
3276 # Audit
3277 audit_upload(req, changelist)
3279 # Done
3280 return SUCCESS_MSG
3283# =============================================================================
3284# Action maps
3285# =============================================================================
3288class Operations:
3289 """
3290 Constants giving the name of operations (commands) accepted by this API.
3291 """
3293 CHECK_DEVICE_REGISTERED = "check_device_registered"
3294 CHECK_UPLOAD_USER_DEVICE = "check_upload_user_and_device"
3295 DELETE_WHERE_KEY_NOT = "delete_where_key_not"
3296 END_UPLOAD = "end_upload"
3297 GET_ALLOWED_TABLES = "get_allowed_tables" # v2.2.0
3298 GET_EXTRA_STRINGS = "get_extra_strings"
3299 GET_ID_INFO = "get_id_info"
3300 GET_TASK_SCHEDULES = "get_task_schedules" # v2.4.0
3301 REGISTER = "register"
3302 REGISTER_PATIENT = "register_patient" # v2.4.0
3303 START_PRESERVATION = "start_preservation"
3304 START_UPLOAD = "start_upload"
3305 UPLOAD_EMPTY_TABLES = "upload_empty_tables"
3306 UPLOAD_ENTIRE_DATABASE = "upload_entire_database" # v2.3.0
3307 UPLOAD_RECORD = "upload_record"
3308 UPLOAD_TABLE = "upload_table"
3309 VALIDATE_PATIENTS = "validate_patients" # v2.3.0
3310 WHICH_KEYS_TO_SEND = "which_keys_to_send"
3313OPERATIONS_ANYONE = {
3314 Operations.CHECK_DEVICE_REGISTERED: op_check_device_registered,
3315 # Anyone can register a patient provided they have the right unique code
3316 Operations.REGISTER_PATIENT: op_register_patient,
3317}
3318OPERATIONS_REGISTRATION = {
3319 Operations.GET_ALLOWED_TABLES: op_get_allowed_tables, # v2.2.0
3320 Operations.GET_EXTRA_STRINGS: op_get_extra_strings,
3321 Operations.GET_TASK_SCHEDULES: op_get_task_schedules,
3322 Operations.REGISTER: op_register_device,
3323}
3324OPERATIONS_UPLOAD = {
3325 Operations.CHECK_UPLOAD_USER_DEVICE: op_check_upload_user_and_device,
3326 Operations.DELETE_WHERE_KEY_NOT: op_delete_where_key_not,
3327 Operations.END_UPLOAD: op_end_upload,
3328 Operations.GET_ID_INFO: op_get_id_info,
3329 Operations.START_PRESERVATION: op_start_preservation,
3330 Operations.START_UPLOAD: op_start_upload,
3331 Operations.UPLOAD_EMPTY_TABLES: op_upload_empty_tables,
3332 Operations.UPLOAD_ENTIRE_DATABASE: op_upload_entire_database,
3333 Operations.UPLOAD_RECORD: op_upload_record,
3334 Operations.UPLOAD_TABLE: op_upload_table,
3335 Operations.VALIDATE_PATIENTS: op_validate_patients, # v2.3.0
3336 Operations.WHICH_KEYS_TO_SEND: op_which_keys_to_send,
3337}
3340# =============================================================================
3341# Client API main functions
3342# =============================================================================
3345def main_client_api(req: "CamcopsRequest") -> Dict[str, str]:
3346 """
3347 Main HTTP processor.
3349 For success, returns a dictionary to send (will use status '200 OK')
3350 For failure, raises an exception.
3351 """
3352 # log.info("CamCOPS database script starting at {}",
3353 # format_datetime(req.now, DateFormat.ISO8601))
3354 ts = req.tabletsession
3355 fn = None
3357 if ts.operation in OPERATIONS_ANYONE:
3358 fn = OPERATIONS_ANYONE.get(ts.operation)
3360 elif ts.operation in OPERATIONS_REGISTRATION:
3361 ts.ensure_valid_user_for_device_registration()
3362 fn = OPERATIONS_REGISTRATION.get(ts.operation)
3364 elif ts.operation in OPERATIONS_UPLOAD:
3365 ts.ensure_valid_device_and_user_for_uploading()
3366 fn = OPERATIONS_UPLOAD.get(ts.operation) # type: ignore[assignment]
3368 if not fn:
3369 fail_unsupported_operation(ts.operation)
3370 result = fn(req)
3371 if result is None:
3372 # generic success
3373 result = {TabletParam.RESULT: ts.operation}
3374 elif not isinstance(result, dict):
3375 # convert strings (etc.) to a dictionary
3376 result = {TabletParam.RESULT: result}
3377 return result
3380@view_config(
3381 route_name=Routes.CLIENT_API,
3382 request_method=HttpMethod.POST,
3383 permission=NO_PERMISSION_REQUIRED,
3384)
3385@view_config(
3386 route_name=Routes.CLIENT_API_ALIAS,
3387 request_method=HttpMethod.POST,
3388 permission=NO_PERMISSION_REQUIRED,
3389)
3390def client_api(req: "CamcopsRequest") -> Response:
3391 """
3392 View for client API. All tablet interaction comes through here.
3393 Wraps :func:`main_client_api`. Handles exceptions.
3395 Internally, replies are managed as dictionaries.
3396 For the final reply, the dictionary is converted to text in this format:
3398 .. code-block:: none
3400 k1:v1
3401 k2:v2
3402 k3:v3
3403 ...
3404 """
3405 # log.debug("{!r}", req.environ)
3406 # log.debug("{!r}", req.params)
3407 t0 = time.time() # in seconds
3409 # -------------------------------------------------------------------------
3410 # Establish session (requires something coherent from the client)
3411 # -------------------------------------------------------------------------
3412 try:
3413 ts = req.tabletsession
3414 except UserErrorException as e:
3415 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e)
3416 return TextResponse(
3417 "Not a valid CamCOPS API request\n", status="400 Bad Request"
3418 )
3420 # -------------------------------------------------------------------------
3421 # Call main API
3422 # -------------------------------------------------------------------------
3423 try:
3424 resultdict = main_client_api(req)
3425 resultdict[TabletParam.SUCCESS] = SUCCESS_CODE
3426 status = "200 OK"
3428 except UserErrorException as e:
3429 log.warning("CLIENT-SIDE SCRIPT ERROR: {}", e)
3430 resultdict = {
3431 TabletParam.SUCCESS: FAILURE_CODE,
3432 TabletParam.ERROR: escape_newlines(str(e)),
3433 }
3434 status = "200 OK"
3436 except ServerErrorException as e:
3437 log.error("SERVER-SIDE SCRIPT ERROR: {}", e)
3438 # rollback? Not sure
3439 resultdict = {
3440 TabletParam.SUCCESS: FAILURE_CODE,
3441 TabletParam.ERROR: escape_newlines(str(e)),
3442 }
3443 status = "503 Database Unavailable: " + str(e)
3445 except Exception as e:
3446 # All other exceptions. May include database write failures.
3447 # Let's return with status '200 OK'; though this seems dumb, it means
3448 # the tablet user will at least see the message.
3449 log.exception("Unhandled exception") # + traceback.format_exc()
3450 resultdict = {
3451 TabletParam.SUCCESS: FAILURE_CODE,
3452 TabletParam.ERROR: escape_newlines(exception_description(e)),
3453 }
3454 status = "200 OK"
3456 # Add session token information
3457 resultdict[TabletParam.SESSION_ID] = ts.session_id
3458 resultdict[TabletParam.SESSION_TOKEN] = ts.session_token
3460 # Convert dictionary to text in name-value pair format
3461 txt = "".join(f"{k}:{v}\n" for k, v in resultdict.items())
3463 t1 = time.time()
3464 log.debug("Time in script (s): {t}", t=t1 - t0)
3466 return TextResponse(txt, status=status)