If I was going to use fiberglass, it would be the mat, not cloth. Cloth works nice for wrapping around corners and edges. In your mind, you just add layer after layer and build it up.
In reality, I've found that whatever you are using to lay down the next layer will stick to the layer below and causes it to move around. You get air bubbles and all kinds of flaws.
Sure, you could do 1 layer at a time, curing in between, but who has time for that?
So I would use mat, as it is thicker and won't move around as much. If you wanted to do mat 1 layer at a time, at least you don't need to do as many layers.
BUT -
I think it boils down to how much time you want to spend, how good (or bad) you want it to look, and how often the roof will get wet in the future.
You mentioned small budget and not much experience. Small budget has been my M.O. for most of my life, but experience follows from that.
For the trim:
If you remove all of the trim at the front and rear glass to do this 'band aid' the 'right' way, you will find more damage, you'll find clips that are rusted beyond re-use (the glass must come out to replace them) and you'll likely bend some of the trim getting it off. That trim is a monster to get off when there's rust involved, even with the right tools, as each clip seems to want a different technique. And even after the clip is released, the rust and crud still hold the trim in place sometimes. It will snowball.
The side trim along the roofrail is a crimp-type fit, with no clips, and is
much easier - I would remove that if you're up to it. Once you get a starting point, you just kinda walk it off the whole length.
Use a stiff rotary brush in a drill to clean the gutter area as much as possible.
For the holes:
If it were my car, I would use a die grinder with a carbide burr and go around the perimeter of each hole as surgically as possible, going 1/8"-1/4" outward into clean metal. If you don't have enough shop air, a decent Makita electric model is ~$100. I have the one shown, and for some jobs it is invaluable. The smaller the diameter of the carbide burr, the more surgical you can be. I would use a pointed-tip burr for this work.
Be careful around the trim, put layers of duct tape on to protect it. If the die grinder grabs and you nick a piece - that's unavoidable. (again, the smaller dia burr helps here)
Keep the shopvac hose nearby and such up as much debris as you can, as you go. Metal shards from a carbide burr are nasty, and you don't want them raining thru the headliner for the next year.
Use a nice sharp-edged scraper around the trim, to get as much of the old vinyl and rustflakes off. The hook on the one shown would be useful, dragging it toward you. As would a heavy-duty curved pick.
For areas that aren't yet perforated, and maybe around the trim - use the rust converter of your choice. Don't be afraid to make a few tiny holes in any questionable areas. Soak the rust converter in around the trim with a small paintbrush. Got to kill as much rust as possible.
After all the holes are hogged out, I would sand the whole roof with a DA sander, with maybe 150 grit. Wouldn't hurt to carefully put a phosphoric metal prep on that - make sure it doesn't run onto your paint. If you don't have shop air, the Porter Cable 6" electric DA is a great tool, I have one with many years of DIY use on it.
Now we're ready to fill.
For holes - peck the edges inward a bit and use fiberglass mat cut to shape, overlaying the hole by 1/4-1/2". It may sag into larger holes, like the one on that C-pillar. Maybe tape it into place to hold the edges, it should peel off after it's all cured? Do a few layers on the larger holes as needed. I would recommend 1/8" thickness. If the hole wraps behind the trim - do the best you can.
For the smaller holes - cut a small plug of fiberglass, roll it into a cork shape, and put it in all the holes. Mix up some resin, and carefully dribble it on/in the plugs. Don't go nuts, to prevent the resin from dripping thru and hitting the headliner. After cured, cut them flush with a hacksaw blade. Or you could use a fiberglass-reinforced bondo on small holes - just pack it in.
Take the DA and do a rough feather-edge at the holes, making sure to not weaken where the fiberglass is adhering to the metal. Do NOT try to make this be like good bodywork. You want the fiberglass to have integrity at the edges.
And maybe let the whole business cook in the sun for a few days to dry any moisture or rust converter left behind.
Spray the whole top with a metal-etching primer. Rustoleum has that in spray-can form at home stores now, 3 cans should do it?
Coating it.
FYI with Flex-Seal - it is great stuff, but it has a waxy surface and paint will NOT stick. I recently repaired some of the bottom edge of my shed's wall sheathing and learned that the hard way. The can warns of this, too, BTW. So use black or white to emulate a VT, I guess?
I would coat the whole top with roll-on pickup bed liner. I would do 2 coats to hide the sins, and I would pay particular attention to get it right in the interface between the roof and the stainless trim for the front and rear glass. Edge it in with a paintbrush if necessary. The side trim is not involved in any water-sealing, so I would put that trim on afterward. (don't coat the lip it attaches to, gotta make sure it will still snap on)
If some of the repaired areas are too noticeable - roll more bedliner onto those areas to thicken it up a little?
Some folks might say it heresy to put the bedliner onto the edge of the trim, but the goal is to keep water from seeping back in those areas. This is not meant to be a pretty repair, but a waterproof one. Around the trim is where the rust is hardest to remove, and therefore needs the most waterproofing. And - it can still breathe from where the trim lays on the glass. So you're not really waterproofing at the trim, you're just making an umbrella for it. Rust tends to be stable if it stays dry.
Overall:
Based on the pics and some similar VT rust repairs I've had to do, I would expect to spend a full weekend doing all of this, perhaps +1 day more.
Depending on what tools you need to buy, and the primer, fiberglass, bedliner, etc - it still might cost $200-500 to do this repair. (but IMO that's a reasonable cost for the benefit)
And after that, I would NOT get that roof wet unless washing the car, and I would not rinse the roof, I would damp-cloth it.
I believe, if you get as much of the rust out as possible, and metal-prep what is still there, that this repair could last for years if you keep the roof dry.