[AI Technique] Lineart -> Flux -> HighRes

[doe.1971]

But all being considered, I need a bigger VRAM card ... using general RAM slows it down tremendously ...

A lot of new nodes I should look into more carefully. And the peculiar case of Comfy-FrontEnd-Package that Comfy says it's too old (<1.16.9) despite the fact that I installed the latest 1.81.1

In any case, many praises for your work and the virgins await you!

[xs70]

To economize on VRAM, try this workflow: https://filebin.net/kg5vupk7b9gleg4s . It's the same as the other workflow but with a GGUF loader node. GGUF versions of Flux are smaller with lower precision.

You will need to download this model: https://civitai.com/models/705823?modelVersionId=789555
Location: ComfyUI/models/unet

Then refresh your ComfyUI window, and select the new model in the first node of group 1: Unet Loader (GGUF)

This model is NSFW and 5GB in size. The controlnet model is 2GB. With a bit of luck, everything will fit in your VRAM. If not, you can give this 3.76GB version a shot: https://civitai.com/models/705823?modelVersionId=789543

GGUF for some reason doesn't work well with LoRA's. I think you can combine 1, after that things get really slow, which defeats the point of using GGUF.

[doe.1971]

Thanks no need, I prefer a larger model trading off longer times. I was just comparing with SDXL or similar 6..7 GB models. But on a whole I'd rather load 16GB if I have the option to spill some in RAM.

I'm not sure if it's possible in Fooocus, I tried once to load a 12 GB model and crashed.

[xs70]

Thanks no need, I prefer a larger model trading off longer times.

Yeah, I'm like that as well. That said, it's only a 5GB download, smaller than an SDXL model. Worth at least giving it a try and see what comes out. Your call.

[doe.1971]

Yeah, I'm like that as well. That said, it's only a 5GB download, smaller than an SDXL model. Worth at least giving it a try and see what comes out. Your call.

It's allright man, you did already all the heavy lifting and I'm REALLY grateful. Beside, I need an incentive to invest 1000 bucks into a piece of hardware to replace something I've bought last year. Damn, the useful life of PC components became shorter and shorter ...

[xs70]

Beside, I need an incentive to invest in a 1000 bucks into a piece of hardware to replace something I've bought last year.

Makes sense though. Flux is worth it.

[mtf]

Thanks for the writeup.

I am no artist, so I took a Gary Roberts picture and experimented.

Original on the far left. The others are a favorite of a bunch of different runs using the same prompt, except adding "photorealistic," "watercolor" and "pencil sketch" respectively. Each has some of the usual AI artifacts, especially around hands. Rope seems to also be something that Gen AI also has problems with.

I've attempted once or twice to use inpainting/outpainting to patch up the artifacts, but so far I haven't gotten good results - likely due to my inexperience and inexpertise.

I also tried playing around with some Bishop artwork, but the controlnet seems to have great difficulty with picking out outlines from his high-contrast style and deciphering faces when they're covered with masks, tape and/or gags.

Still, much fun to be had!

[xs70]

Nice. Second one is my favorite.

[LLL]

This procedure explains how to generate highres Flux images starting from line art, ie. black lines on a white background.

If you could make a step by step guide of what to install, from where, what buttons to press to generate AI based on a drawing or other image, I'm sure there will be a chorus of appreciative voices around here.

I do have a 8GB RTX 4060 Ti on a Ryzen 9, 16 cores, 128 GB RAM. I lack the guiding hand to make it churn stuff.

The hardest part is installing ComfyUI. Once it is installed, copy the workflow file, below, onto your local machine, then drag and drop it on top of a pre-opened ComfyUI window. You may be prompted to install missing custom nodes (hit yes). If some nodes are still unrecognized, go to the ComfyUI Manager Menu and click Install Missing Custom Nodes.

I use an RTX 4060 Ti with 16GB of VRAM (= memory on the graphics card), with 32GB of system memory. Not bad, not too fancy either. Flux requires a lot of VRAM. This workflow will work on lower spec machines, to a certain point; in cases of insufficient VRAM, ComfyUI will try to load part of the model in system memory. This works but image generation will be slower. If a PC doesn't have enough VRAM and it doesn't have enough system memory either, ComfyUI will throw an insufficient memory error.

Workflow file: https://filebin.net/svtmfphl88wgyzxz

Result (input, output):


Lana01b.jpg



lana_highres_00008_.jpg

A likely stupid question, which will definitely earn me the ire of his JDness, but...

Can all this be done in Fooocus? Is ComfyUI something similar? An alternative? AN integration?

Before installing something else I would like to know if it is potentially conflicting with Fooocus, since I barely learned to use that, or if it is another similar tool.

[xs70]

Not a stupid question.

Stable Diffusion is at its core a Python library program (a collection of libraries). Python is a command line programming language, and not everyone wants to write code to create (GIMP) art.

Various smart people have come up with their own solutions (ComfyUI, Fooocus, Krea, ...) to make Stable Diffusion more user friendly. The trick lies in designing a web interface which presents the user with buttons, checkboxes, flow diagrams, etc.: visual interfaces that let the user create a Stable Diffusion workflow without having to resort to programming. When the user then hits Queue, or Generate, or Execute, or whatever the final button is called, the web interface translates their settings into corresponding Python commands; sends these commands to the Stable Diffusion system; and returns the resulting image via the web interface. Rinse and repeat.

There's a tradeoff between user friendliness and the amount of options one exposes to the user. The more choices the user has to make, the more complicated the workflow gets. ComfyUI workflows can get pretty complicated. Fooocus is on the user friendly side of the spectrum: it's set up really well for standard Stable Diffusion workflows (prompt -> highres -> inpaint) yielding great results. But it may not have support for more complicated stuff, like ControlNet, which this workflow relies on. I did a quick google search and while I didn't find anything specific to ControlNet, I found this explanation:

With alternatives like AUTOMATIC1111’s Web UI, ComfyUI, MidJourney, PixArt Alpha, and DALL·E 3, it’s essential to understand where Fooocus carves out its niche. Designed by Illyasviel, the developer behind ControlNet, Fooocus targets low-code users who want a streamlined image generation experience—without the complexity of systems like A1111 or ComfyUI.

Long story short: I don't know if this works in Fooocus, it depends on whether Fooocus supports Canny ControlNet. I use ComfyUI to make images that Stable Diffusion won't generate easily, like GIMP art. Some of the most impressive (non GIMP) Stable Diffusion images I've seen came out of Fooocus, so there's nothing wrong with it. Just fewer possibilities, but those that are implemented do work great. The two platforms shouldn't interfere with each other; afaik you can install both on the same machine.