Commit Graph

68 Commits

Author SHA1 Message Date
Dominik Agres 0d7ac417c8 Datasets 2025-05-04 09:30:57 +02:00
Dominik Agres be25b1e77f Reqs update 2025-05-04 09:30:04 +02:00
agres 709ad41b88 reqs 2025-04-09 13:59:58 +02:00
agres 4f724ea20b Moved important file 2025-04-09 13:54:42 +02:00
agres 771026b224 feature extraction 2025-04-09 13:40:17 +02:00
agres c919befbdb Restructure files 2025-04-09 13:37:26 +02:00
agres d91eed279d Song data csv files 2025-04-09 13:30:23 +02:00
agres 33deb35a6c updated gitignore 2025-04-09 13:28:02 +02:00
agres b47bb40dbc Based code 2025-04-06 18:18:57 +02:00
agres 372e98ded9 Audoi analysis of tracks 2025-04-03 17:28:46 +02:00
agres 4706332180 Analysis files 2025-04-03 15:14:53 +02:00
agres 4ae2a5ace6 reworked db initialisation to separate a test and prod db 2025-03-24 12:20:37 +01:00
agres dbb80e207b Now the connection witch is initialised is cleanly closed by the handler 2025-03-24 10:12:03 +01:00
agres 589f82345a Added debug function for verbose conole logging 2025-03-24 10:11:16 +01:00
agres 9febe63643 Added crash logging, extended the --export variable to define size of exorted data, setup verbose logging 2025-03-24 10:10:09 +01:00
agres aae9115b23 added a function witch sets the console handler to DEBUG level. this can be used for verbose logging 2025-03-24 10:09:13 +01:00
agres 10a490bf9c Logging refactor in auth 2025-03-23 23:27:28 +01:00
agres afe6a563ef Refactored logger 2025-03-23 23:22:41 +01:00
agres db983cfc8f Refactored to own logger 2025-03-23 23:19:12 +01:00
agres 87f84b250e Logging and documentation in scraper 2025-03-23 23:16:26 +01:00
agres f714200225 Extended logging and documentation for api handler 2025-03-23 23:05:29 +01:00
agres c65a1a8c8b Unified spotify api request into one function 2025-03-23 21:56:09 +01:00
agres 9d48f7bef7 Fixed counter bug 2025-03-23 21:55:38 +01:00
agres 7552d43702 Bigger batch size 2025-03-23 21:55:11 +01:00
agres e73752c015 Fixed counter issue where the accumilation would be calculated wrongly and the requests were not at the fullest capacity 2025-03-23 21:54:44 +01:00
agres 164ea9aca9 'fix' for a bug 2025-03-23 20:49:34 +01:00
agres af6b3dba13 The most convoluted, code-efficient, bloated, boilerplated, duplicated unneccecary undocumented, soon to be regretted code 2025-03-23 20:49:00 +01:00
Chris Kiriakou 625f09cfdf Change data-docker dir in code-block description 2025-03-23 19:38:01 +01:00
Chris Kiriakou 99a19d38d6 Fix stupid typo that costed dome 5 hours to debug. 2025-03-23 19:35:49 +01:00
Dominik ff9d726b47 Feat/import gdrp data (#26)
* Some simple code for extracting data from the jsons

* Jupiter Notebook

* Mac specific gitignore

* Fixed finding paths to floders

* Delete src/gdpr_data directory

* Updated gitignore to include my testing file

* Added the standard saving path for the database in the database handler, this way multiple files dont have to be updated when moving database position

* Moved the API usage wrappers into an own file, added a function for getting multiple track_ids at once, this still needs to be tested more

* Further code for extracting data from the gdpr files

* Forgor

* Final&Tested version of get_multiple_tracks_information endpoint

* Further functionality: The code now extracts the id of each listened song and makes a api call to get info about these songs via the multiple tracks api. Furthermore we track the songs witch the call is made for already and skip these

* Added function to map catalouged ids into the play history

* Added args parser to runtime program, cleaned up some code

* Fixed a bug where the database would always try to create tables, eaven if it exists

* Added some small text for clean interface

* Some final fixes to actual code, fixed db bug, reversed the order of database entries

* Some documentation

* Added -export args to docker runtime

* fix
2025-03-23 18:48:57 +01:00
Dominik 535225392f Merge pull request #24 from agresdominik/feat/containerize_app
Feat/containerize app
2025-03-22 10:06:02 +01:00
Chris Kiriakou 71e9db8738 Remove unused tag. 2025-03-21 19:17:12 +01:00
Chris Kiriakou fd1d8e6874 Change the directory location 2025-03-21 19:13:46 +01:00
Chris Kiriakou 0531a346b5 Add new folders 2025-03-21 19:12:44 +01:00
Chris Kiriakou aeb7c49068 Create improved Dockerfile with volume mounts for storing data
* Add volume mounts to Dockerfile to enable persisten storing of
  database and tokens when using docker
* Makefile helps with cleanup and building the Dockerfile
* Startup fires the predictify runtime inside the container
* `.dockerignore` contains files that are not needed by the docker,
  e.g. files that would otherwise slow build process down
2025-03-21 19:07:30 +01:00
Chris Kiriakou 3922c7640e Add documentation for docker and small visual improvements 2025-03-21 19:05:00 +01:00
Chris Kiriakou 38e0c69bea Move .env file to config dir, change db path for improved project structure
* Change path of db and token to `data` which helps to sperate source
  code from data files like `tokens.json` & `spotify_scraped.db`
* Change path of `.env` `src/env/` to `config/` for the same reason as
  mentioned above
* Added the newly created files to `.gitigore` accordingly
2025-03-21 18:57:13 +01:00
Chris Kiriakou 2d4e951693 Move Dockerfile & startup script to sperate directory 2025-03-21 18:54:07 +01:00
agres cf7ede5495 Fix startup 2025-03-19 21:05:33 +01:00
Dominik e6e342ea8e Merge pull request #22 from agresdominik/staging
Staging to Main
2025-03-19 20:55:56 +01:00
Dominik 29deb2c971 Merge pull request #21 from agresdominik/feat/containerize_app
Create a custom Dockerfile to decouple application dependencies from …
2025-03-19 20:55:03 +01:00
Chris Kiriakou 3fe19fd41f Fix Dockerfile. 2025-03-19 20:54:13 +01:00
Dominik 1ad803bbfb Merge pull request #19 from agresdominik/feat/database_handler
Feat/database handler
2025-03-19 19:55:03 +01:00
agres 601b94e405 Added timer to avoid working with cronjobs 2025-03-19 19:54:06 +01:00
agres 6b35a34826 refacor & simple logger 2025-03-19 19:38:22 +01:00
agres 7db5c51867 Big commit 2025-03-19 18:20:49 +01:00
Chris Kiriakou ae477c01a4 Merge branch 'staging' into feat/containerize_app 2025-03-19 17:55:35 +01:00
Dominik e574c6bc00 Feat/auth multiple tokens integration (#15)
* Implemented a multiple token read/write/refresh function for the authentificator

* Rework auth with simple authentificate
2025-03-19 17:44:17 +01:00
Chris Kiriakou 9b9002e751 Create a custom Dockerfile to decouple application dependencies from host.
* Create a dockerfile based on alpine.
* Create a start script to startup scraper.
2025-03-19 16:56:23 +01:00
Dominik 57ac860f1f Merge pull request #11 from agresdominik/feat/scraping_implementation
Feat/scraping implementation
2025-03-19 16:31:39 +01:00