Page MenuHomeSealhub

Re-enable the local disk file cache option, manage max cache size
Closed, ResolvedPublic3 Points

Description

ImageRouter used to have it's own caching mechanizm, and it worked quite well! The only thing it was missing was clearing up old cached images, so the disk usage went up pretty quickly. We tried mitigating it with nginx cache, but it's notoriously tricky to set up, clears its cache seemingly without reason, causing the website to slow down as many images are re-generated.

This task is about bringing back the caching functionality to the way it was, meaning:

  1. If someone requests an image that is already currently being converted, don't run a new conversion thread, wait for the currently running to finish and return its result
  2. Store all the resized and converted images on the HDD and use them instead of resizing and converting a new image on each request

The above can be relatively easily achieved by reverting some of the changes from the commit that took away the caching functionality: https://hub.sealcode.org/rRIMAGEROUTER872653e19b9f5f9c9f6ec9d6e8dd5500e094a804

Additionally, the caching now needs to be a little bit more sophisticated:

  1. There needs to be a parameter to the ImageRouter that would tell how many gigs of HDD space can the cache take up
  2. Every time there's a new cached image added to the cash storage, a cleanup action must run that checks if the total amount of hdd space used by the cache exceeds the limit set up in point 1. If it does, the action will remove a certain amount of least recently accessed files (see https://www.howtogeek.com/517098/linux-file-timestamps-explained-atime-mtime-and-ctime/) so the total amount of used cache storage comes down just below the limit
  3. The action from point 2 does not run concurrently. If someone adds a new image to the cache while the action is running, it doesn't get executed another time. We avoid running two or more instances of the action from point 2 in parallel
  4. The action described in point 2 does not block the http response for the request that triggered it. The action starts running only after closing HTTP request

Details

Tested by
kuba-orlik
Coded by
luryxyt
Reviewed by
kuba-orlik
Action required from
luryxyt
PHID of the last column the processing script has seen this task in
PHID-PCOL-rt4p72bfiw2fnfkxc5fl
Timestamp of the time this task got moved into the current column
1692894775016

Revisions and Commits

Restricted Differential Revision

Event Timeline

kuba-orlik triaged this task as Very High priority.Jul 9 2023, 18:41
kuba-orlik created this task.
jenkins-user set Timestamp of the time this task got moved into the current column to 1688920875151.Jul 9 2023, 18:41
jenkins-user set PHID of the last column the processing script has seen this task in to PHID-PCOL-2ksh7xlolko4ggqo7rez.
jenkins-user changed Timestamp of the time this task got moved into the current column from 1688920875151 to 1692894775016.Aug 24 2023, 18:32
jenkins-user changed PHID of the last column the processing script has seen this task in from PHID-PCOL-2ksh7xlolko4ggqo7rez to PHID-PCOL-rt4p72bfiw2fnfkxc5fl.
jenkins-user added a project: Unknown Object (Project).Aug 29 2023, 19:40
kuba-orlik lowered the priority of this task from Very High to High.Sep 4 2023, 17:07
luryxyt moved this task from To do to Doing on the Kanban board.
jenkins-user updated Action required from, added: luryxyt.
luryxyt added a revision: Restricted Differential Revision.Oct 5 2023, 19:00
jenkins-user moved this task from Doing to Review on the Kanban board.
jenkins-user updated Action required from, added: KamilBaczkowski, prostyfranek, Wirusik, wrzesinski-hubert, FilipI, brtkuu; removed: luryxyt.
jenkins-user added a subscriber: luryxyt.
jenkins-user moved this task from Review to Doing on the Kanban board.
jenkins-user updated Reviewed by, added: kuba-orlik.
jenkins-user updated Action required from, added: luryxyt; removed: KamilBaczkowski, prostyfranek, Wirusik, wrzesinski-hubert, FilipI, brtkuu.
jenkins-user moved this task from Doing to Review on the Kanban board.
jenkins-user updated Action required from, added: kuba-orlik; removed: luryxyt.
jenkins-user moved this task from Review to Doing on the Kanban board.
jenkins-user updated Action required from, added: luryxyt; removed: kuba-orlik.
kuba-orlik shifted this object from the Restricted Space space to the S5 Publiczna space.Oct 30 2023, 13:22
kuba-orlik changed the visibility from "All Users" to "Public (No Login Required)".
luryxyt added a revision: Restricted Differential Revision.Feb 9 2024, 18:19
jenkins-user moved this task from Doing to Review on the Kanban board.
jenkins-user updated Action required from, added: kuba-orlik; removed: luryxyt.
jenkins-user moved this task from Review to Doing on the Kanban board.
jenkins-user updated Action required from, added: luryxyt; removed: kuba-orlik.
jenkins-user moved this task from Doing to Review on the Kanban board.
jenkins-user updated Action required from, added: kuba-orlik; removed: luryxyt.
jenkins-user assigned this task to luryxyt.
jenkins-user moved this task from Review to Ready for testing on the Kanban board.
jenkins-user updated Action required from, added: AmeliaB, Koossa, karolwojtowicz87, brtkuu, prostyfranek, zbartosz, radoptak; removed: kuba-orlik.
jenkins-user updated Tested by, added: kuba-orlik.
jenkins-user moved this task from Ready for testing to Ready to land on the Kanban board.
jenkins-user updated Action required from, added: luryxyt; removed: AmeliaB, Koossa, karolwojtowicz87, brtkuu, prostyfranek, zbartosz, radoptak.
jenkins-user moved this task from Ready to land to Review on the Kanban board.
jenkins-user updated Action required from, added: kuba-orlik; removed: luryxyt.
jenkins-user moved this task from Review to Ready for testing on the Kanban board.
jenkins-user moved this task from Ready for testing to Ready to land on the Kanban board.
jenkins-user updated Action required from, added: luryxyt; removed: kuba-orlik.
kuba-orlik added a commit: Restricted Diffusion Commit.Mar 2 2024, 17:39