Add --web-snapshot-viewer mode: Playwright-based screenshot proxy for old browsers

[maagenta]

Hello,

I have been working on a new feature for WebOne. The idea is to allow browsing the modern web in old browsers using screenshots. When the browser sends a request, the proxy navigates to the same webpage using Playwright, takes screenshots, and returns an HTML page with the screenshots.

This only applies when the proxy is started with the --web-snapshot-viewer flag. Otherwise the proxy starts normally.

I took care to keep the changes to existing proxy files minimal. All the snapshot viewer logic is contained in the web-snapshot-viewer folder.

Here is a demo running on Mac OS 10.3 with Safari 1:
https://youtu.be/-AfQm8GCOt8

You can find the architecture documentation here:
https://github.com/maagenta/webone/blob/web-snapshot-viewer/web-snapshot-viewer/ARCHITECTURE.md

The code is in the web-snapshot-viewer branch, which I created from the main branch:
https://github.com/maagenta/webone/tree/web-snapshot-viewer

[maagenta]

[atauenis]

Hello. This is a nice feature (and seems that is good implemented), but it makes the app more closer to Web-Rendering Proxy than a regular proxy server. However this enables correct rendering and website working, it does not allowing text selection, copy&paste, file download, and is slow.

Also I'm not sure about true system requirements of the feature. WebOne is designed for all systems starting from $50 Win8 netbooks/nettops (or Linux VPS/VDS or Raspberry Pi 1-gen and up) or old MacBooks, to modern $1000+ hardware. WRP feature seems to be more for last category.

Probably it will be great for a fork of WebOne, but not for main release line.

Or... Hm... This feature may be neccessary for anti-bot challenges. Many websites are detecting WebOne as a bot, and are showing CAPTCHAs that are using complex JS things not available in old browsers. And using a latest browser is a requirement. But only CloudFlare are registering IP in white list after successfull pass. ReCaptcha, Yandex Captcha are using cookies and probably some other more, so passing a test on external browser is not enough.

[maagenta]

Thanks for your answer.

1. Text selection: The idea is to implement that in the future. I think I can create a script that puts overlaid text in a hidden div, so you can copy & paste.

2. Download Files: It can be implemented using a script in the proxy and in the client's script.

3. System Requirements: That is a real issue. I configured the proxy to handle a minimum of 1,000 threads, which is a lot. Also, it has to take the screenshots, divide them into chunks and compare each chunk using MD5. All the images and MD5 hashes are stored in RAM to make it faster, which obviously consumes a lot of RAM. I was thinking about the user would port the proxy to a home server or to the cloud, so the computation would not be done by the client's computer itself. Even without this feature, I think that's the smartest thing to do. I think it's not a good idea to run webone proxy in a Windows 98 laptop with 64mb of ram.

4. Captchas: That's a real issue. I was thinking about sending screenshots in real time using MutationObserver, and also implementing an update of the client's viewport when an event is triggered (click, keydown, scroll, etc.). In that case, the captcha can be resolved directly in the client.

The main problem with adding these kinds of features is that old computers can't handle a large number of HTTP requests quickly, nor render that many images.

Please take into account that this proxy should only be enabled if the user uses the web-snapshot-viewer flag. Without the flag, it should be disabled.

But I can understand that you don't want that feature in your proxy, as it might cause it to lose focus.

I think it could be a good idea to create my own repository, or to find an existing web rendering proxy and try to improve it.

[atauenis]

1. Text selection: The idea is to implement that in the future. I think I can create a script that puts overlaid text in a hidden div, so you can copy & paste.

Like in OCR-ed documents (PDF, DJVU). Combines both graphic display and native text layer. In time of MSIE5 era it were possible via a div with 1% opacity (if do with 0%, MSIE hides the div). May be useful for Safari 1.x too.
This way also used in 2000-s for copy-protected websites, where an div with copyright message overlapped contents.

2. Download Files: It can be implemented using a script in the proxy and in the client's script.

I know that the feature is already implemented in Browservice. And the implementation is good. It using a pop-up window that handles the upload process.

I think it's not a good idea to run webone proxy in a Windows 98 laptop with 64mb of ram.

For a more-or-less comfortable work currently an Core 2 Duo or dual-core Atom with 2 GB and basic VGA 640x480 is enough for WebOne. This is a performance of cheap virtual hosting providers too (except that they have no graphics at all). Previously WebOne worked on 32-bit Windows 7 on even slower machines, but after removing Win7 support, it is no more. Both amd64 Linux and Win8.1 utilizes more resources, so requirements became a bit larger.