best image classifications for 8vram

Posted by ashendonep@reddit | LocalLLaMA | View on Reddit | 17 comments

i have rtx 3060ti 8vram , trying to use it to classify images like : 'out of 5k car images tell me which ones is red ' i tried with

the best one was qwen3.5:9b but also the slowest one (like 3 minutes per image ) , so having 5k images takes a decade , what can i do because ai did not help ToT

here is my options if it help

options: {
        num_gpu: -1,
        num_ctx: 4096,
        temperature: 0,
        top_k: 1,
        top_p: 1,
        repeat_penalty: 1,
        use_mlock: false,
        use_mmap: true,
        flash_attn: true,
        kv_cache_type: "q4_0",
        num_keep: 0,
      },
      keep_alive: 120,
    });