report.html

Report generated on 02-Jun-2021 at 15:36:53 by pytest-html v3.1.1

Environment

JAVA_HOME C:\Program Files\Java\jre1.8.0_271
Packages {"pluggy": "0.13.1", "py": "1.10.0", "pytest": "6.2.4"}
Platform Windows-10-10.0.18362-SP0
Plugins {"html": "3.1.1", "metadata": "1.11.0"}
Python 3.6.10

Summary

7 tests ran in 107.46 seconds.

7 passed, 0 skipped, 0 failed, 0 errors, 0 expected failures, 0 unexpected passes

Results

Result Test Duration Links
Passed tests/test_table_user_log.py::test_table_user_log_run 12.82
-----------------------------Captured stdout setup------------------------------
Going to create spark session
-----------------------------Captured stderr setup------------------------------
21/06/02 15:35:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
------------------------------Captured stdout call------------------------------
######### Read logs into a dataframe data/test/log_sample.log.gz ######### Select all required fields to store into userlog ######### User_id definition ######################################## ######### Additional features ##################################### ######### Add Bucketing ########################################## Size log 1001 ######### Writing on disk with partition of ipHashBucket 0 1 loggedtimestamp 2015-07-22T09:00:28.019143Z 2015-07-22T09:00:27.824580Z loggeddate 2015-07-22 2015-07-22 hour 9 9 minute 0 0 elb marketpalce-shop None sourceIP 123.242.248.130 203.91.211.44 request_processing_time 0.000022 None backend_processing_time 0.026109 None elb_status_code 200 None backend_status_code 200 None received_bytes 0 None sent_bytes 699 None URL https://paytm.com:443/shop/authresponse?code=f... None user_agent Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKi... None os Windows device Other browser Chrome user_id 123.242.248.130 203.91.211.44 sourceIPPrefix 123.242.248 203.91.211 useragenthash 217164113 42 ts 1437588028 1437588027 IPHashBucket 25 25
------------------------------Captured stderr call------------------------------
[Stage 2:> (0 + 1) / 1] [Stage 3:> (0 + 1) / 1] 21/06/02 15:35:20 WARN Utils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.debug.maxToStringFields' in SparkEnv.conf.
Passed tests/test_table_user_session_log.py::test_table_user_session_log 30.73
-----------------------------Captured stdout setup------------------------------
Going to execute table_user_session_log_run ###### Generate user_session_id, is_new_session from user_log ####################### ['loggedtimestamp', 'hour', 'minute', 'elb', 'sourceIP', 'request_processing_time', 'backend_processing_time', 'elb_status_code', 'backend_status_code', 'received_bytes', 'sent_bytes', 'URL', 'user_agent', 'os', 'device', 'browser', 'user_id', 'sourceIPPrefix', 'useragenthash', 'ts', 'IPHashBucket', 'loggeddate'] 0 1 user_id 1.187.217.224 1.187.217.224 loggedtimestamp 2015-07-22T09:00:28.784397Z 2015-07-22T09:00:29.938427Z ts 1437588028 1437588029 sourceIP 1.187.217.224 1.187.217.224 user_agent Mozilla/5.0 (Linux; U; Android 4.4.2; en-US; A... Mozilla/5.0 (Linux; U; Android 4.4.2; en-US; A... useragenthash 236440475 236440475 URL https://paytm.com:443/shop/authresponse?code=8... https://paytm.com:443/shop/cart os Android Android device Aqua Star 5.0 Aqua Star 5.0 last_event None 2015-07-22T09:00:28.784397Z loggeddate 2015-07-22 2015-07-22 hour 9 9 IPHashBucket 8 8 user_session_id 1.187.217.224_2015-07-22T09:00:28.784397Z 1.187.217.224_2015-07-22T09:00:28.784397Z ###### Statistics per user_session : sessionduration, start, end, ... ################### 0 1 user_session_id 14.141.115.46_2015-07-22T09:00:29.935999Z 1.39.35.172_2015-07-22T09:00:28.563787Z user_id 14.141.115.46 1.39.35.172 sourceIP 14.141.115.46 1.39.35.172 IPHashBucket 1 4 starttimestamp 2015-07-22T09:00:29.935999Z 2015-07-22T09:00:28.563787Z endtimestamp 2015-07-22T09:00:30.266162Z 2015-07-22T09:00:28.563787Z session_duration 1 0 n_unique_url 2 1 n_events 2 1
-----------------------------Captured stderr setup------------------------------
21/06/02 15:35:23 WARN FileUtil: Failed to delete file or dir [D:\_devs\Python01\gitdev\arepo\zesresr\output\test\tables\usersession\_temporary]: it still exists. [Stage 39:> (0 + 1) / 8] [Stage 39:=======> (1 + 1) / 8] [Stage 39:==============> (2 + 1) / 8] [Stage 39:=====================> (3 + 1) / 8] [Stage 39:=============================> (4 + 1) / 8] [Stage 39:====================================> (5 + 1) / 8] [Stage 39:===========================================> (6 + 1) / 8] [Stage 39:==================================================> (7 + 1) / 8]
Passed tests/test_table_user_session_log.py::test_table_user_session_log_stats 16.95
------------------------------Captured stderr call------------------------------
[Stage 61:======================================> (70 + 1) / 100] [Stage 63:=====================================================> (98 + 1) / 100] [Stage 65:==========> (2 + 1) / 11] [Stage 65:===============> (3 + 1) / 11] [Stage 65:====================> (4 + 1) / 11] [Stage 65:=========================> (5 + 1) / 11] [Stage 65:===============================> (6 + 1) / 11] [Stage 65:====================================> (7 + 1) / 11] [Stage 65:==============================================> (9 + 1) / 11] [Stage 65:==================================================> (10 + 1) / 11] [Stage 66:===============> (3 + 1) / 11] [Stage 66:====================> (4 + 1) / 11] [Stage 66:=========================> (5 + 1) / 11] [Stage 66:===============================> (6 + 1) / 11] [Stage 66:=========================================> (8 + 1) / 11] [Stage 66:==============================================> (9 + 1) / 11] [Stage 66:==================================================> (10 + 1) / 11] [Stage 68:===============> (3 + 1) / 11] [Stage 68:=========================> (5 + 1) / 11] [Stage 68:===============================> (6 + 1) / 11] [Stage 68:====================================> (7 + 1) / 11] [Stage 68:=========================================> (8 + 1) / 11] [Stage 68:==================================================> (10 + 1) / 11] [Stage 69:===============> (3 + 1) / 11] [Stage 69:====================> (4 + 1) / 11] [Stage 69:=========================> (5 + 1) / 11] [Stage 69:===============================> (6 + 1) / 11] [Stage 69:====================================> (7 + 1) / 11] [Stage 69:=========================================> (8 + 1) / 11] [Stage 69:==============================================> (9 + 1) / 11] [Stage 72:===============> (3 + 1) / 11] [Stage 72:====================> (4 + 1) / 11] [Stage 72:=========================> (5 + 1) / 11] [Stage 72:====================================> (7 + 1) / 11] [Stage 72:=========================================> (8 + 1) / 11] [Stage 72:==============================================> (9 + 1) / 11] [Stage 73:====================> (4 + 1) / 11] [Stage 73:=========================> (5 + 1) / 11] [Stage 73:===============================> (6 + 1) / 11] [Stage 73:====================================> (7 + 1) / 11] [Stage 73:=========================================> (8 + 1) / 11] [Stage 73:==============================================> (9 + 1) / 11] [Stage 75:===============> (3 + 1) / 11] [Stage 75:====================> (4 + 1) / 11] [Stage 75:===============================> (6 + 1) / 11] [Stage 75:====================================> (7 + 1) / 11] [Stage 75:=========================================> (8 + 1) / 11] [Stage 75:==================================================> (10 + 1) / 11] [Stage 76:==========> (2 + 1) / 11] [Stage 76:====================> (4 + 1) / 11] [Stage 76:=========================> (5 + 1) / 11] [Stage 76:===============================> (6 + 1) / 11] [Stage 76:====================================> (7 + 1) / 11] [Stage 76:=========================================> (8 + 1) / 11] [Stage 76:==============================================> (9 + 1) / 11]
Passed tests/test_table_user_session_stats.py::test_table_user_session_stats 24.76
-----------------------------Captured stdout setup------------------------------
Going to execute table_user_session_stats_run ###### Load usersessionstats table ###### 1) Get the avg, min and max session duration whole users. +--------------------+--------------------+--------------------+ |min_session_duration|max_session_duration|avg_session_duration| +--------------------+--------------------+--------------------+ | 0| 2| 0.2743764172335601| +--------------------+--------------------+--------------------+ None ###### 2) Get Unique URL count per session +--------------------+------------+ | user_session_id|n_unique_url| +--------------------+------------+ |223.196.243.127_2...| 3| |122.180.141.102_2...| 1| |203.200.232.121_2...| 3| |122.169.170.254_2...| 2| |203.123.182.161_2...| 1| |122.177.211.114_2...| 1| |111.119.231.194_2...| 1| |114.143.246.178_2...| 1| |117.205.160.216_2...| 1| |121.244.199.142_2...| 1| +--------------------+------------+ only showing top 10 rows None ###### 3) Most Engaged : users with longest session duration +---------------+--------------------+--------------------+--------------------+ | user_id|min_session_duration|max_session_duration|avg_session_duration| +---------------+--------------------+--------------------+--------------------+ | 203.91.192.5| 2| 2| 2.0| |117.240.248.194| 2| 2| 2.0| | 14.102.53.58| 2| 2| 2.0| | 115.248.71.233| 2| 2| 2.0| | 122.15.156.144| 2| 2| 2.0| |106.216.149.189| 2| 2| 2.0| | 59.94.7.155| 2| 2| 2.0| | 117.211.83.18| 2| 2| 2.0| | 114.31.251.116| 2| 2| 2.0| | 170.74.56.80| 2| 2| 2.0| +---------------+--------------------+--------------------+--------------------+ only showing top 10 rows Top 10 most engaged users None
-----------------------------Captured stderr setup------------------------------
[Stage 81:===============> (3 + 1) / 11] [Stage 81:====================> (4 + 1) / 11] [Stage 81:===============================> (6 + 1) / 11] [Stage 81:====================================> (7 + 1) / 11] [Stage 81:=========================================> (8 + 1) / 11] [Stage 81:==================================================> (10 + 1) / 11] [Stage 83:===============> (3 + 1) / 11] [Stage 83:====================> (4 + 1) / 11] [Stage 83:===============================> (6 + 1) / 11] [Stage 83:====================================> (7 + 1) / 11] [Stage 83:=========================================> (8 + 1) / 11] [Stage 83:==================================================> (10 + 1) / 11] [Stage 86:==========> (2 + 1) / 11] [Stage 86:====================> (4 + 1) / 11] [Stage 86:=========================> (5 + 1) / 11] [Stage 86:====================================> (7 + 1) / 11] [Stage 86:=========================================> (8 + 1) / 11] [Stage 86:==============================================> (9 + 1) / 11] [Stage 88:===============> (3 + 1) / 11] [Stage 88:====================> (4 + 1) / 11] [Stage 88:=========================> (5 + 1) / 11] [Stage 88:===============================> (6 + 1) / 11] [Stage 88:====================================> (7 + 1) / 11] [Stage 88:=========================================> (8 + 1) / 11] [Stage 88:==============================================> (9 + 1) / 11] [Stage 92:===================> (1 + 1) / 3] [Stage 92:======================================> (2 + 1) / 3] [Stage 93:==========> (2 + 1) / 11] [Stage 93:===============> (3 + 1) / 11] [Stage 93:====================> (4 + 1) / 11] [Stage 93:=========================> (5 + 1) / 11] [Stage 93:===============================> (6 + 1) / 11] [Stage 93:====================================> (7 + 1) / 11] [Stage 93:=========================================> (8 + 1) / 11] [Stage 93:==============================================> (9 + 1) / 11] [Stage 93:==================================================> (10 + 1) / 11]
Passed tests/test_table_user_session_stats.py::test_table_user_session_stats_ip 9.24
------------------------------Captured stderr call------------------------------
[Stage 115:===================> (2 + 1) / 6] [Stage 115:============================> (3 + 1) / 6] [Stage 115:======================================> (4 + 1) / 6] [Stage 116:===================> (2 + 1) / 6] [Stage 116:============================> (3 + 1) / 6] [Stage 116:===============================================> (5 + 1) / 6] [Stage 118:============================> (3 + 1) / 6] [Stage 118:======================================> (4 + 1) / 6] [Stage 119:======================================> (4 + 1) / 6] [Stage 119:===============================================> (5 + 1) / 6] [Stage 122:======================================> (4 + 1) / 6] [Stage 123:============================> (3 + 1) / 6] [Stage 123:======================================> (4 + 1) / 6] [Stage 125:============================> (3 + 1) / 6] [Stage 125:===============================================> (5 + 1) / 6] [Stage 126:===================> (2 + 1) / 6] [Stage 126:============================> (3 + 1) / 6] [Stage 126:======================================> (4 + 1) / 6]
Passed tests/test_utils.py::test_config_load 0.00
No log output captured.
Passed tests/test_utils.py::test_spark_check 9.31
------------------------------Captured stdout call------------------------------
0 1 ipaddr 1.187.217.224 1.187.236.169 ts 1437588028 1437588030 duration 100 200 0 1 ipaddr 1.187.217.224 1.187.236.169 ts 1437588028 1437588030 duration 100 200 +-------------+----------+--------+ |ipaddr |ts |duration| +-------------+----------+--------+ |1.187.217.224|1437588028|100 | |1.187.236.169|1437588030|200 | +-------------+----------+--------+ only showing top 2 rows
------------------------------Captured stderr call------------------------------
[Stage 132:> (0 + 1) / 1] [Stage 141:> (0 + 1) / 1] [Stage 150:> (0 + 0) / 1][Stage 151:> (0 + 1) / 1] [Stage 150:> (0 + 1) / 1]