Images aren't loaded properly by two providers - how to disable them? #3856

se7entynine · 2024-09-27T23:39:48Z

se7entynine
Sep 27, 2024

How did you install SearXNG?

proxmox →
docker compose - searxng-docker - 2024.9.23+6be56aee1 →
traefik v3.1.4 →
cloudflare → public facing website

What happened?
Not all pictures are rendered during a search. Search term isnt relevant to this bug. Browser logs show mostly images from live.staticflickr.com and *.mm.bing.net arent loaded properly, but I deactivated bing in the image search provider tab. Is there another way to block these providers?

How To Reproduce
Search for anything and then check the image tab and/or look if pictures were loaded with either live.staticflickr.com or bing.net .

Expected behavior
All pictures are loaded correctly.

Screenshots & Logs

Docker logs:

2024-09-24 16:46:59,193 ERROR:searx.webapp: Exception on /image_proxy [GET]

Traceback (most recent call last):

  File "/usr/lib/python3.12/site-packages/flask/app.py", line 1473, in wsgi_app

    response = self.full_dispatch_request()

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/flask/app.py", line 882, in full_dispatch_request

    rv = self.handle_user_exception(e)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/flask/app.py", line 880, in full_dispatch_request

    rv = self.dispatch_request()

         ^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/flask/app.py", line 865, in dispatch_request

    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/searxng/searx/webapp.py", line 1059, in image_proxy

    resp, stream = http_stream(method='GET', url=url, headers=request_headers, allow_redirects=True)

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/searxng/searx/network/__init__.py", line 258, in stream

    response = next(generator)  # pylint: disable=stop-iteration-return

               ^^^^^^^^^^^^^^^

  File "/usr/local/searxng/searx/network/__init__.py", line 228, in _stream_generator

    raise obj_or_exception

  File "/usr/local/searxng/searx/network/__init__.py", line 195, in stream_chunk_to_queue

    async with await network.stream(method, url, **kwargs) as response:

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__

    return await anext(self.gen)

           ^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpx/_client.py", line 1573, in stream

    response = await self.send(

               ^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpx/_client.py", line 1617, in send

    response = await self._send_handling_auth(

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpx/_client.py", line 1645, in _send_handling_auth

    response = await self._send_handling_redirects(

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects

    response = await self._send_single_request(request)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpx/_client.py", line 1719, in _send_single_request

    response = await transport.handle_async_request(request)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpx/_transports/default.py", line 353, in handle_async_request

    resp = await self._pool.handle_async_request(req)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 262, in handle_async_request

    raise exc

  File "/usr/lib/python3.12/site-packages/httpcore/_async/connection_pool.py", line 245, in handle_async_request

    response = await connection.handle_async_request(request)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpcore/_async/connection.py", line 92, in handle_async_request

    raise exc

  File "/usr/lib/python3.12/site-packages/httpcore/_async/connection.py", line 69, in handle_async_request

    stream = await self._connect(request)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpcore/_async/connection.py", line 149, in _connect

    stream = await stream.start_tls(**kwargs)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 78, in start_tls

    raise exc

  File "/usr/lib/python3.12/site-packages/httpcore/_backends/anyio.py", line 69, in start_tls

    ssl_stream = await anyio.streams.tls.TLSStream.wrap(

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/site-packages/anyio/streams/tls.py", line 132, in wrap

    await wrapper._call_sslobject_method(ssl_object.do_handshake)

  File "/usr/lib/python3.12/site-packages/anyio/streams/tls.py", line 172, in _call_sslobject_method

    raise EndOfStream from None

anyio.EndOfStream

[pid: 28|app: 0|req: 183/1598] 192.168.224.1 () {66 vars in 2934 bytes} [Tue Sep 24 16:46:59 2024] GET /image_proxy?url=https%3A%2F%2Flive.staticflickr.com%2F3612%2F3382159981_e3b73140e5.jpg&h=c00deda7780482d6331a6af191ecf24b6545a777df168a106d2c574b6cd73093 => generated 265 bytes in 39 msecs (HTTP/1.1 500) 8 headers in 289 bytes (1 switches on core 1)

Browser logs:

GET
https://REDACTED/image_proxy?url=https://live.staticflickr.com/7004/6494613787_9a3031fa7d_b.jpg&h=ed7f5e62663386ce21da3316659225fec71f9c00f56fb3b8e8ba982adbf3c184
[HTTP/3 500  880ms]

	
GET
	https://REDACTED/image_proxy?url=https://live.staticflickr.com/7004/6494613787_9a3031fa7d_b.jpg&h=ed7f5e62663386ce21da3316659225fec71f9c00f56fb3b8e8ba982adbf3c184
Status
500
VersionHTTP/3
Übertragen1,86 kB (265 B Größe)
Referrer Policyno-referrer
Anfrage-PrioritätLow
DNS-AuflösungSystem

    	
    cf-cache-status
    	DYNAMIC
    cf-ray
    	REDACTED-FRA
    content-security-policy
    	default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; form-action 'self' https://github.com/tiekoetter/searxng/issues/new; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; connect-src 'self'; img-src 'self' data: https://*.tile.openstreetmap.org; frame-src 'self' https://www.youtube-nocookie.com https://player.vimeo.com https://www.dailymotion.com https://www.deezer.com https://www.mixcloud.com https://w.soundcloud.com https://embed.spotify.com https://open.spotify.com/
    content-type
    	text/html; charset=utf-8
    date
    	Tue, 24 Sep 2024 20:28:52 GMT
    permission-policy
    	accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()
    priority
    	u=4,i
    referrer-policy
    	same-origin
    server
    	cloudflare
    server-timing
    	total;dur=37.412, render;dur=0
    strict-transport-security
    	max-age=15552000; includeSubDomains; preload
    x-content-type-options
    	nosniff
    x-download-options
    	noopen
    x-frame-options
    	SAMEORIGIN
    x-robots-tag
    	none,noarchive,nosnippet,notranslate,noimageindex,noindex,nofollow
    x-xss-protection
    	1; mode=block
    	
    Accept
    	image/avif,image/webp,image/png,image/svg+xml,image/*;q=0.8,*/*;q=0.5
    Accept-Encoding
    	gzip, deflate, br, zstd
    Accept-Language
    	de,en-US;q=0.7,en;q=0.3
    Alt-Used
    	REDACTED
    Connection
    	keep-alive
    Cookie
    	authentik_proxy_C7AVdJyL=EBPTVGQWC6VJGBJCN_REDACTED_vzc2qZI9A
    DNT
    	1
    Host
    	REDACTED
    Priority
    	u=4, i
    Sec-Fetch-Dest
    	image
    Sec-Fetch-Mode
    	no-cors
    Sec-Fetch-Site
    	same-origin
    Sec-GPC
    	1
    TE
    	trailers
    User-Agent
    	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:130.0) Gecko/20100101 Firefox/130.0

Additional context

Docker compose:

version: "3.7"

services:
  searxng-redis:
    container_name: searxng-redis
    image: docker.io/valkey/valkey:7-alpine
    command: valkey-server --save 30 1 --loglevel warning
    restart: unless-stopped
    networks:
      - searxng
    volumes:
      - ${PWD}/valkey-data2:/data
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
      - DAC_OVERRIDE
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"

  searxng:
    container_name: searxng
    image: docker.io/searxng/searxng:latest
    restart: unless-stopped
    networks:
      - searxng
    ports:
      - "7070:8080"
    volumes:
      - ${PWD}/searxng:/etc/searxng:rw
    environment:
      - SEARXNG_BASE_URL=https://${SEARXNG_HOSTNAME:-localhost}/
      - TZ=Europe/Berlin
      
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"
    env_file:
      - .env

networks:
  searxng:

Traefik:

http:
  routers:
    searxng: 
      entryPoints:
      - "https"
      rule: "Host(`REDACTED`)"
      middlewares:
        - error-pages
        - searxng-secure-headers
      tls: 
        options: {default}
      service: searxng
      
  services:
    searxng:
      loadBalancer:
        servers:
          - url: "http://REDACTED:7070"
        passHostHeader: true
        
  middlewares:
    searxng-secure-headers:
      headers:
        accessControlAllowMethods: ["GET", "OPTIONS", "PUT"]
        accessControlMaxAge: 100
        stsSeconds: 63072000
        stsIncludeSubdomains: true
        stsPreload: true
        forceSTSHeader: true
        customFrameOptionsValue: "SAMEORIGIN" #CSP takes care of this but may be needed 
        contentTypeNosniff: true 
        browserXssFilter: true 
        referrerPolicy: "same-origin" 
        contentsecuritypolicy: "default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; form-action 'self'; font-src 'self'; frame-ancestors 'self'; base-uri 'self'; connect-src 'self'; img-src 'self' data: https://*.tile.openstreetmap.org; frame-src 'self' https://www.youtube-nocookie.com https://player.vimeo.com https://www.dailymotion.com https://www.deezer.com https://www.mixcloud.com https://w.soundcloud.com https://embed.spotify.com https://open.spotify.com/"
        featurePolicy: "camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';" 
        customResponseHeaders:
          X-Robots-Tag: "none,noarchive,nosnippet,notranslate,noimageindex,noindex,nofollow"
          server: ""
          Permission-Policy: "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=()"

Answered by return42

Sep 28, 2024

We recommend not to use cloudflare / its the cause of various problems in the setups and Cloudflare can read everything (because they MITM your https connection) .. which contrasts with any efforts to preserve privacy. The SearXNG core team therefore does not provide any support for questions regarding such misguided undertakings. Thank you for your understanding

View full answer

return42 · 2024-09-28T05:59:12Z

return42
Sep 28, 2024
Maintainer

We recommend not to use cloudflare / its the cause of various problems in the setups and Cloudflare can read everything (because they MITM your https connection) .. which contrasts with any efforts to preserve privacy. The SearXNG core team therefore does not provide any support for questions regarding such misguided undertakings. Thank you for your understanding

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Images aren't loaded properly by two providers - how to disable them? #3856

Uh oh!

{{title}}

Uh oh!

Replies: 1 comment

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Images aren't loaded properly by two providers - how to disable them? #3856

Uh oh!

se7entynine Sep 27, 2024

Replies: 1 comment

Uh oh!

return42 Sep 28, 2024 Maintainer

se7entynine
Sep 27, 2024

return42
Sep 28, 2024
Maintainer